- what produces a number
pistra guardrails eval, the binary that enforces- what is scored
- the pipeline, not the checkpoint on the Hub
- on what
- labelled cases at byte offsets, sets pinned by digest
- what is published
- the models rejected, beside the ones shipped
pistra ships some models and pins other people's. Both get measured before they go in, against the traffic a deployment actually carries rather than the sets on a model card. Often the number we needed did not exist anywhere, and the run that produced it is the report.
How a report is made
Every number comes from pistra guardrails eval, the same binary and the same code path that enforces. The command takes a gateway config, a file of labelled cases and the types to score, then presents each case as one chat turn through the gateway's own inspection and scores what came out against what a person marked, at byte offsets into the text as sent.
That is a measurement of the pipeline rather than of the checkpoint, which is where these numbers part company with the ones beside a model on the Hub. A model ships with a tokenizer binding, a window scheme, a script fold, a decoder and a threshold. Every one of those can lose a span the model found or land it on the wrong bytes, and a score computed in a notebook describes a model nobody is running.
Each report names its evaluation sets and their sha256, and the build every model ran on. Where the same sentences exist in several languages, the run is an invariance sweep read against a reference language, English by default: the table says which rows a model keeps, loses and gains against that reference, so a guard that holds in English and fires on everything in Arabic shows the drop instead of an average.
What a report here is held to
- Negative results are published. A model we rejected, a technique that cost more than it returned, a clean-looking score that turned out to mean silence. Those are the reports worth writing, and leaving them out is how a measurement becomes marketing.
- Reports are re-issued, not frozen. A blog post describes the day it was written and is never edited. A report is a measurement, so when a model, a set or the harness moves, it is re-run and carries both dates.
- The run is yours to repeat. The sets are named, the digests are printed and the command is documented, so a report can be disagreed with on its own numbers rather than taken on trust.
What we do not publish
No corpus text. A report names every evaluation set and its digest, and the mismatched cases stay local, because they are the corpora's text and not ours to redistribute. No ranking of vendors we have not measured on the same build. And no claim about a language we have not run, which is why several reports end by naming the languages they could not characterise.
The reports
The most recent is Arabic NER corpora put the clitic inside the name. Every report is listed together, newest first.