Run a detector in shadow mode

A new detector that is wrong denies real requests. mode and schedule are separate axes so you can introduce one without betting an outage on it.

• mode, does policy read this detector’s findings? inline or shadow.
• schedule, does the request wait for it? sync, overlap or async.

All six combinations are meaningful. Walk up them.

1. Observe for free

Add this configuration:

    - type: remote
      name: ner
      mode: shadow
      schedule: async

Nothing waits and policy ignores it. Findings are recorded and counted:

pistra_guardrail_findings_total{detector="ner",entity,mode="shadow"}

Compare against the detector you already trust. You are looking for entities it finds that nothing else does, and for entities it claims where the text is innocuous.

2. Measure what it would actually cost

Add this configuration:

      mode: shadow
      schedule: overlap

Now the response waits for it, but policy still ignores it. This is the only way to learn the added latency, because it is the difference between the detector and the provider’s round trip, not the detector’s own runtime.

Watch pistra_guardrail_scan_seconds and the overall pistra_overhead_seconds. If added latency is near zero, the detector is free in the position you would actually run it.

3. Let it act

Add this configuration:

      mode: inline
      schedule: overlap

Policy now reads it. Start with a rule that annotates rather than denies, then tighten:

  rules:
    - name: ner-observation
      when: 'annotations.exists(a, a.detector == "ner")'
      action: annotate

4. Only if it must never leave the building

Add this configuration:

      mode: inline
      schedule: sync

The request now waits before anything is forwarded, and you pay the detector’s full latency. This is the only setting where the provider never sees the content.

Require two detectors to agree

Available once more than one detector runs. An overlap pass runs the whole ensemble, not only the late members. The cheap ones just ran, so the delta cache answers for them. That makes agreement expressible:

guardrails:
  agreement_boost: 0.2
  rules:
    - name: both-or-nothing
      when: >
        annotations.exists(a, a.detector == "pii" && a.entity_type == "US_SSN") &&
        annotations.exists(a, a.detector == "ner" && a.entity_type == "US_SSN")
      action: deny

agreement_boost adds to an annotation’s score when a second detector independently annotates an overlapping span with the same entity type.

Two detectors agreeing is the strongest corroboration available and the most expensive, it costs a second detector. Within one detector there are two cheaper ones, and a rule can ask for them by name:

annotations.exists(a, a.entity_type == "US_SSN" &&
                      (a.validated || a.context_supported))

validated is a checksum that accepted the span. context_supported is one of the recognizer’s context words found beside it. Prefer these to a score threshold, they say what you mean, and they do not go stale when a recognizer is retuned. See PII coverage.

Fail open or closed, on purpose

Add this configuration:

      on_error: fail_closed    # fail_open | fail_closed

A fail_closed detector that errors returns 503 guardrail_error. A fail_open one is skipped. Neither is right in general. A shadow detector should always be fail_open.

Related

• Schedules
• Run a slow detector off the request path