Why the number comes from the binary

A model on a hub carries precision and recall on its card. Those numbers are about the checkpoint.

What a rule acts on comes out of four stages after it, and every one of the four can lose a span the model found.

Four stages, each of them correct

The tokenizer splits the request into vocabulary items, the ones this checkpoint was trained on and no others. The window scheme cuts a sequence past max_tokens into overlapping windows, 512 tokens with a stride of 14 by default, because a model with a fixed input length has to be given a long turn in pieces. The decoder folds subword scores back into words, groups the BIO tags into spans, and lands each span on byte offsets in the text as sent. The threshold turns a score into a verdict.

Each of the four is right for its own job. The tokenizer is the one the model was exported with, and any other produces ids the model never saw. The windowing keeps a long segment scorable and gives every position at least one window where it is not at an edge. The decoder recovers the offsets a redactor needs to rewrite exactly the bytes it meant to. The threshold picks an operating point on a curve.

Together they are a system the notebook never ran. A token classifier computes logits and stops. Everything after that is pipeline code, written in Python by HuggingFace, inherited by Presidio when it calls that pipeline, and inherited by nobody who calls the graph directly. pistra writes that half in Go. So the thing being scored has to be the whole of it, or the score is about a model no deployment contains.

The number that moved without the model

The clearest case we have came out of the Gulf names set, and the model was not involved.

XLM-R uses a SentencePiece tokenizer, which does not split trailing punctuation off a word. الشمري? arrives as one word of two tokens: the surname, and the question mark. The default aggregation is max, meaning the word takes the label of its most confident subword. The question mark scored higher than the name, labelled nothing, and the word was dropped.

The fix is in the decoder. Punctuation at either edge of a word is now made a word of its own before any aggregation runs, which is what a WordPiece tokenizer would have done in the first place. On the tag slices of gulf-names.jsonl, exact F1 for a name inside an English sentence went from 0.57 to 1.00, and a name after a title from 0.90 to 1.00.

The checkpoint is byte for byte the same in both runs. Two scores moved by a wide margin, one of them from failing to perfect, and the cause was a rule about where a word ends. A token-level F1 cannot see it, because a token-level F1 never lands a span on bytes and so never has to decide what a word is.

The one it does not fix

Measuring the whole pipeline also tells you when the pipeline is not the problem.

On the clitic slice of the same set, both models return the name with the prefixed letter attached: ومحمد where the set marked محمد. Exact F1 is 0.228 for mBERT and 0.246 for XLM-R, against 0.75 to 0.80 on the rest of the set. Overlap-allowed precision on that slice is exactly 1.000 for both. Every span they returned contains the whole name.

Nothing is misaligned. The decoder produced one contiguous span of the right type at high confidence, on the boundary the model predicted. Both models learned Arabic from ANERcorp, which is annotated a token per line, and Arabic clitics attach inside the token, so a clitic-prefixed name is one token whose tag covers the clitic. Two architectures with different tokenizers and different vocabularies agree to within 1.4 points on which names come back one letter long, because they learned the same convention.

That is a finding about the corpus, not about the harness, and the fix belongs after the decoder rather than in it. The point here is that the two cases look identical in a notebook and are opposites in a deployment. One is a pipeline defect that a checkpoint swap would not have touched. The other is a model doing exactly what it was taught, which no decoder change will undo. The gap between the exact and the overlap-allowed columns is what separates them, and that gap only exists once the spans are landing on bytes.

What the harness is

pistra guardrails eval takes labelled text through the same inspection a request gets. The config is compiled the way the gateway compiles it, the models are fetched and verified against their digests, the native libraries load, the thresholds apply, and each case is presented as one chat turn. What came out is scored against what a person marked, at byte offsets into the text as sent, exact and with overlap allowed, per type and per tag.

A tag on a case names a slice, and the report repeats its totals for each, which is how one file carries every hazard a detector faces and the number for each stays visible. mixed and title moved and the rest of the table held still. That is only legible because the slices were separate before anybody knew which one would move.

What the method costs

A report is worth exactly as much as the harness it came from, so when the harness moves the report has to be taken again. The decoder change above retired every number taken before it. Those numbers were right about the pipeline that produced them, and that pipeline no longer exists. A reader comparing two rows has to know that one thing between them changed and nothing else did.

That is why each report names the build every model ran on and the sha256 of every set, and why a report is re-issued with both dates rather than frozen the way a post is. It is also why disagreeing with one is expensive: it takes the binary, the config, the models and the sets, and then it takes running them.

A number that costs nothing to produce and nothing to check is a claim wearing a decimal point. The expense is what makes the other kind worth reading.

All posts