Skip to content
pistra

Run a slow detector off the request path

A transformer NER model takes hundreds of milliseconds. So does a judge asking a chat model. The provider takes hundreds of milliseconds too, and those waits do not have to happen one after the other.

Overlap the detector with the upstream call

Section titled “Overlap the detector with the upstream call”

Add this configuration:

guardrails:
  detectors:
    - type: remote
      name: ner
      schedule: overlap
      timeout: 2s
      remote:
        endpoint: http://presidio-analyzer:3000
        offsets: chars

schedule: overlap starts the detector when the request is forwarded and joins before the first byte goes back to the client. Added latency is max(0, detector − provider latency): a 590 ms scan against an 800 ms time-to-first-token costs nothing.

The join happens at the last possible moment, after the response body is read for a unary call, and after the first frame for a stream. A refusal is then still an ordinary 403 rather than an error smuggled into a stream that already started.

Decide what a redact rule means there

Section titled “Decide what a redact rule means there”

Overlap concedes one thing. The provider saw the bytes. A redact rule that fires on an overlap pass cannot do what it says, so choose which way to be wrong:

guardrails:
  overlap:
    on_redact: deny     # deny | allow
  • deny refuses the response. The client gets nothing rather than something policy wanted changed.
  • allow forwards it and records the finding.

Either way it increments pistra_guardrail_overlap_redact_total{route,outcome}. A non-zero value means a rule is not doing what its author wrote, alert on it.

Keep schedule: sync when the requirement is that the provider never sees the content at all.

Stop waiting entirely

Section titled “Stop waiting entirely”

Add this configuration:

guardrails:
  detectors:
    - type: remote
      name: deep-scan
      schedule: async
  async:
    max_in_flight: 4
    timeout: 30s

async waits for nothing, not the request, not the response. It still enforces, one turn later. Findings land in the delta cache keyed by segment text, and a conversation replays its history on the next turn. The ordinary sync pass reads them from the cache and denies or redacts having waited for nothing.

The cost, stated plainly:

  • The turn where text first appears is governed only by the detectors that were waited for.
  • Single-shot traffic, an embedding call, a one-off completion, has no next turn, so async there is audit only.
  • Past max_in_flight a scan is dropped and counted in pistra_guardrail_async_total{route,disposition}, never queued, because work whose value is being current is worthless once it is late.

An async detector is named in Result.Deferred, so an audit record can never imply it ran when it did not.

Section titled “Related”