A guard model deployed against Arabic traffic is scored in English. That is how its card reports it and how it is compared. The report Guard models are reported in English ran protectai/deberta-v3-base-prompt-injection-v2 over topics.jsonl, ordinary customer turns in English and Arabic. It catches the Arabic jailbreak turns eight out of eight and flags 80.2 percent of the ordinary Arabic turns as an attack. On a deny rule a false positive is a refused customer, and that is the number that breaks a deploy.
The reason is in the tokenizer. Arabic costs that model 3.18 subword pieces per word, and a sequence of unfamiliar tokens looks like an attack. Switching to a model with a multilingual vocabulary cuts the false-positive rate from 80.2 percent to 1.2 percent, and also cuts the English F1 from 0.914 to 0.815 and the Arabic jailbreak turns caught from 8 of 8 to 6 of 8. Which is better depends on the language mix, and no public set measures the half that is missing, recall on Arabic attacks. No public Arabic prompt-injection set exists.
What an evaluation set measures
An evaluation set answers one question: does the detector get the right answer on this text. A prompt-injection set carries attack text that should be flagged, customer text that should pass, and hard negatives that are security-adjacent but harmless. Hard negatives matter most. A request to “ignore the previous instructions” is an injection. A request to “summarise in bullet points, ignoring formatting” is not. A customer asking “where do I change my password” talks about instructions but is not giving one. A detector that fires on all three at the same rate has shown nothing.
The set measures only what is in it: whether the detector works on the words a person wrote, in the script they wrote them, at byte offsets into the text as sent.
Why Arabic attacks need their own set
A guard model learns the language of its training data. It learns what “ignore” sounds like in English, and it learns that a sequence of tokens that do not occur in English training probably means something is wrong. An injection in Arabic has its own shape, its own vocabulary, its own structure, and a false positive rate measured in English or on a translated set does not predict the rate in Arabic.
The research post measured MultiJail, which carried the same 315 red-team prompts translated into ten languages. Every one is a harmful request and none is a prompt injection. On Arabic, ProtectAI flagged 239 of 315. On English, zero. That gap comes from the script. The harm is the same in both. An invariance test, where the same content in two languages should get the same verdict, shows where a detector’s decision depends on the language instead of the meaning. A detector that fires on nothing is also perfectly invariant, so read invariance with recall in the other hand.
How the set is built
One file serves a classifier and a judge. A classifier needs injection against the rest. A judge reading a harm policy needs harmful against the rest. The set carries three classes, injection, harmful, and benign, and the converter maps those onto whatever labels a config declares. One file, one labelling, three uses.
Every row is a parallel pair: the same meaning in Arabic and English, and the verdict should be the same in both. A detector that flags the English injection but misses the Arabic one, or fires on the Arabic customer text and not the English equivalent, shows up as a split row in the invariance table.
Every row records where its words came from. authored means a native speaker writing in the register. drafted-reviewed means a model’s draft that a person rewrote or accepted, and translated-reviewed the same for a translation of a public row. machine is a draft nobody has reviewed, and a published set carries none of them. A source column names the corpus or the generator, so a reader can weigh a row by where it came from.
Hard negatives outnumber attacks by design. A security vocabulary is a technical vocabulary: “bypass”, “token”, “permission”, “prompt”, “instruction”. A request like “bypass the intro screen” or “write a permission prompt for this API” uses that vocabulary without giving an instruction. A document labelled “Instructions for using this tool” does the same. A set that catches that difference matters more than one that only catches attacks.
The leakage rule holds that no text in the set may appear in any training corpus the team uses. A set that borrowed from a model’s training data would measure its memory rather than its judgement. A test in the repository checks every row against every training file.
The plan
The budget table in the sets README lists what the first published version will hold: ninety groups of direct injections, ten per pattern, from override to split; thirty groups of indirect injections, with a document or tool result carrying the attack; sixty harmful rows that a judge should refuse but an injection classifier should pass; ninety hard negatives that talk about instructions without giving any; sixty plain customer turns; and 200 ordinary Arabic instructions sampled from CIDAR, an open Arabic instruction corpus, as plain traffic that carries no attack.
A group is two rows: the same meaning in Arabic and English. 660 authored rows plus 200 corpus rows is the size when no machine draft remains, and the first milestone is smaller: fifty hard negatives and fifty direct injections, reviewed, before any other row is written.
As of 4 September 2026, the set holds 56 groups, 112 rows, all marked as machine: three groups per injection pattern, four indirect, twelve hard negatives, six plain customer turns, five harmful. None has been reviewed, and the set is not published.
What it does and does not do
The set measures whether a detector works on Arabic attack text and Arabic customer text. It does not measure how to deploy one, which threshold to choose, or whether redaction breaks the workflow. It does not carry every dialect: the first rows are Gulf Arabic, dialect:afb. A published set says what it covers, so a reader knows whether the score applies to their traffic.
It measures recall only once its rows are reviewed. Every row is a machine draft today, so a run over them says what the detector does on those words and nothing about recall. Recall needs gold that a person wrote.
How to run it
When the set is published, it is a row per case, in the same form every other evaluation set uses. A reader who wants to measure their own guard model or config runs pistra guardrails eval with the config and the set. The how-to Measure a detector covers the command, the tags, and the -invariance flag, which reads whether the verdict depends on the script instead of the meaning.
Publishing an evaluation set means someone else can run it and check the claim. Until then it stays a draft in the repository, and this post says so.
As of 4 September 2026
pistra is building an Arabic prompt-injection evaluation set with parallel pairs in Arabic and English, an origin tag on every row, and a leakage rule that no text appears in any training corpus the team uses. The first milestone is fifty hard negatives and fifty direct injections, reviewed. As of this date, 56 groups (112 rows) are drafted as machine and none are reviewed. The set is not published. To measure a detector once it is, use pistra guardrails eval.