A model refuses a harmful request more reliably in English than in a language its alignment data barely covers. The paper “Multilingual Jailbreak Challenges in Large Language Models” (Deng, Zhang, Pan and Bing, ICLR 2024) reports the unsafe answer rate rising as the language’s resources fall, and names two scenarios. Unintentional: a user writes in their own language. Intentional: an attacker translates a harmful request to get past safeguards trained in English.
The guard placed in front of the model has the same defect from the same root. A guard is a prompt-injection classifier, a small model trained on one question: does this text try to override the assistant. It is trained on English data, released with English benchmarks, and scored on English sets. So the language where the model is least safe is the language where its guard is least reliable.
A widely deployed guard fires on four in five Arabic customers
The report Guard models are reported in English ran three open prompt-injection classifiers over ordinary customer turns in English and Arabic, and over the same red-team prompts in ten languages. The customer set, topics.jsonl, holds 89 turns in each language: 81 ordinary turns across billing, tech support, account and medical, and 8 tagged as jailbreaks. One of the three, protectai/deberta-v3-base-prompt-injection-v2, is one of the most widely deployed open guard models, and the one pistra had pinned.
On ordinary Arabic customer turns it flags 80.2 percent as an injection. On the same turns in English it flags 1.2 percent. It catches the Arabic jailbreak turns eight out of eight, as it catches the English ones, so it is not deaf to the language. It reads ordinary Arabic customers as attackers.
On a deny rule the request stops before the model sees it. A customer asking about an invoice gets a refusal. At eight in ten, the guard is an outage.
The defect is visible at language boundaries
The report scored each guard on MultiJail, the same 315 red-team prompts annotated into ten languages by the paper’s own annotators. The ten columns of a row mean the same thing, so whatever the right verdict is, it is the same verdict in each language. A verdict that changes between them is a defect that needs no ground truth to see.
None of the 315 is a prompt injection, so for an injection classifier every fire is a false positive. They are harmful requests, a different class, which the previous post in this series takes apart. The report ran the guards over all 3,150 cases (315 prompts times 10 languages) and asked how many verdicts changed between languages.
The ProtectAI model fired on 239 of the 315 Arabic rows, 294 of the Thai rows and 310 of the Bengali rows. It fired on 38 Italian rows, 12.1 percent, so the defect reaches Latin scripts. On English it fired zero times.
One guard, Meta’s Llama Prompt Guard 2 86M, produced zero fires across all 3,150 cases. A verdict that never changes with the language looks perfect on the invariance metric. The metric measures consistency. It says nothing about whether the verdict is ever INJECTION. A model that fires on nothing agrees with itself in every language.
The cause is the tokenizer vocabulary
ProtectAI’s model has a vocabulary of 128,001 entries. 347 of them contain an Arabic character. The model was trained to recognise attacks by the sequence of subword fragments it saw. When Arabic is split into 3.18 fragments per word instead of 1.52, the pattern of unfamiliar pieces looks like an attack.
Swapping to a model whose vocabulary has 7,408 Arabic entries takes the false-positive rate on Arabic customers from 80.2 percent to 1.2 percent, which is English’s rate. The report names the cost: on the one English set neither model trained on, F1 falls from 0.914 to 0.815, and the Arabic jailbreak turns caught fall from 8 of 8 to 6 of 8.
Measure before you pin
Before pinning a guard, measure it on the traffic you carry. An English benchmark says how the guard performs on English and nothing about the rest.
The tool for this is Measure a detector. Its -invariance flag reads the verdict across languages. The set is one case per line, the same meaning in every language, each case named by set, row and language. Run pistra guardrails eval -invariance over it with the config you plan to deploy, and the report gives fired counts and rates per language. That shows where the guard is silent and where it fires on everything.
A guard silent in a language has no recall there. A guard that fires on everything in a language has no precision there, and nothing reaches the assistant. A benchmark measured on English sees neither.
Which guard to ship stays your decision. pistra runs the one you pin and measures it through the same inspection path a request takes, so the number describes your deployment. Measure before you pin.
As of 4 September 2026
The guard-models-and-language report, dated 2026-08-31, measured protectai/deberta-v3-base-prompt-injection-v2, proventra/mdeberta-v3-base-prompt-injection and Llama Prompt Guard 2 86M on two evaluation sets: MultiJail in ten languages, and topics.jsonl, ordinary customer turns and tagged jailbreaks in English and Arabic. The Arabic findings are per the report's table: 239 of 315 MultiJail rows (75.9 percent), 80.2 percent of ordinary Arabic customer turns on topics.jsonl. Vocabulary entries and tokens per word per the report's table. The measurement goes through the gateway's own inspection path, per the pistra docs of the same date. See Guard models are reported in English for the full table, all three guards, and the other languages where the effect is strong.