Test your rules before they ship

Access rules and guardrail rules are the part of a pistra config that is written by hand and read by a machine. A typo in a CEL condition is refused at load. A condition that compiles and means the wrong thing is not, it is found by the first request it gets wrong.

A rule suite finds it in the pull request instead. It is a YAML file of cases, a request as the rules see it, and what they must decide. The cases run against the config’s compiled rules by the same binary that serves them, with nothing listening, no detector run and no provider contacted.

Write the suite

Next to the deployment document, a pistra.tests.yaml:

access_rules:
  - name: research keys are allowed
    key: {name: alice, metadata: {team: research}}
    model: claude-3-5
    expect: allow
  - name: a key without a team is refused, not skipped
    key: {name: svc}
    model: claude-3-5
    expect: error
    rule: research-only
  - name: a key with another team is denied
    key: {name: bob, metadata: {team: sales}}
    model: claude-3-5
    expect: deny
    rule: everyone-else
  - name: a profile's own rule is reached even past a global allow
    key: {name: carol, profile: contractors}
    route: embeddings
    expect: deny
    rule: no-embeddings
  - name: deletes are denied
    at: mcp
    server: github
    method: tools/call
    tool: delete_repo
    expect: deny
    rule: no-deletes
guardrails:
  - name: cards are masked
    key: {name: alice, metadata: {team: research}}
    annotations:
      - {entity_type: CREDIT_CARD, score: 0.95, ref: messages.1.content}
    expect: redact
    rule: mask-cards

A case that gives its key a profile is decided through both rule layers, in the order the runtime uses them: the deployment-wide list first, the profile’s own second. That is the point of testing it here. An allow in the first list ends that list and does not skip the second, and a suite that concatenated them would report the wrong verdict on the configs worth checking. Naming a profile the config does not define fails the case. A case whose caller is an agent or a user rather than a key states the profile the issuer’s binding chose on the case itself, as profile, and is decided through both layers the same way.

Each case states the request the way the rules read it, the same variables the conditions use, and an expectation:

`expect`	means	`rule`
`allow`	no deny rule matched, or an allow rule matched first	none
`deny`	this deny rule matched	required
`error`	this rule could not be evaluated and is `fail_closed`	required
`redact`	guardrails only: this redact rule selected a finding	required
`annotate`	guardrails only: this annotate rule matched	required

key is the virtual key as the key variable sees it. Leave it out for “no key”. An access-rule case is at one hop, at: llm by default, mcp or a2a, and states that hop’s variables: model, provider and route, where provider is a name or a mapping with the metadata a rule reads, provider: {name: azure-ksa, metadata: {residency: sa}}; server, method, tool and args; peer, method, role, task_id, context_id and parts. A variable of another hop fails the case, since no rule at this hop could have read it. The tool, prompt or resource named goes under tool, it is the rules’ name variable, renamed because name is the case’s own.

Guardrail cases state their findings rather than producing them. An annotations list is what a detector would have reported, and the rules decide over it as they would in traffic. category defaults to pii, kind to user (or output on side: response), and ref to the first user message. validated: true states that a checksum confirmed the finding and context_supported: true that a context word raised its score, which is how a case reaches a rule written as a.validated || a.context_supported. That keeps the suite free of detector assets and model downloads. It tests the policy, which is the part you wrote.

A rule that the config does not have is a failure of the case, not a pass by accident, a deny you removed should be found here.

Run it

Use the following example:

$ pistra -config pistra.yaml -test-rules pistra.tests.yaml
ok    access_rules at llm: research keys are allowed
ok    access_rules at llm: a key without a team is refused, not skipped
ok    access_rules at llm: a key with another team is denied
FAIL  access_rules at llm: this one is wrong on purpose
      want allow
      got  deny everyone-else
ok    access_rules at mcp: deletes are denied
5 cases, 1 failed
pistra: 1 of 5 rule cases failed

The exit status is non-zero when any case fails. -config takes the node file, whose deployment is followed, or the deployment document itself. The config goes through the same Build the server runs, the same validation, the same compiled rules, and nothing else. A config that would not load does not get as far as the first case.

Variables the config interpolates from the environment still have to be set. An unset ${VAR} is refused on purpose, here as at startup. A pipeline gives them placeholder values:

- run: pistra -config pistra.yaml -test-rules pistra.tests.yaml
  env:
    ANTHROPIC_API_KEY: placeholder
    ADMIN_TOKEN: placeholder

What an `error` verdict is telling you

A condition that reads a key that is not there, key.metadata.team on a key with no team, args.force on a call without one, does not evaluate. By default the rule is fail_closed and the request is refused with policy_error. The evaluator’s warning on stderr names the rule and the missing key. If that is not what you meant, the fix is a guard in the condition, not fail_open:

condition: 'has(key.metadata.team) && key.metadata.team == "research"'

fail_open is for a rule you would rather have skipped than have refuse, and the suite lets you write that down. A case with expect: allow for a key without the field pins that the rule is skipped, and a case with expect: error pins that it is not.

Preflight and the suite

POST /v1/preflight answers “what would happen to this request” for one live request against a running gateway, including the failover chain and the channel it would take. The suite answers it for a config in a pull request, for every case you thought to write. They read the same rules. Use the suite before merging and preflight when a client asks.

Preflight on the data plane answers only for the key presented. That serves a developer and not an operator. When the client asking is a tenant reporting a refusal you cannot reproduce, ask on their behalf:

curl -sX POST https://gw.example.com/admin/v1/preflight \
  -H "authorization: Bearer $ADMIN_TOKEN" \
  -d '{
        "key": "team-a-prod",
        "path": "/v1/chat/completions",
        "body": {"model": "claude-sonnet-5", "messages": [], "tool_choice": "required"}
      }'

The key is named, never presented, so you do not need the tenant’s credential. Minting one onto their profile to reproduce a refusal is editing the configuration to ask a question about it. Nothing happens as a result: no budget is reserved, no detector runs, no upstream is contacted, and nothing is counted against the key.

Everything the resolution decides comes back as a verdict with a 200, including a refusal the real request would have received, a revoked or suspended key, a denying rule, a model the key may not use. A status means your question was wrong, not that somebody would be refused: 404 for a key the deployment does not have, 422 for a path it does not route, 403 for a caller admin policy does not permit. Cedar scopes it by the key being asked about, so a support role can be given exactly one tenant:

@id("support-sees-team-a")
permit(principal in Role::"support", action == Action::"preflightRequest",
       resource == Resource::"keys/team-a-prod");

Guardrails are out of scope on both planes. Scanning means real detector calls and, on the async schedules, effects that outlive the request. A dry run must not have those. Preflight answers the routing and fidelity question and does not pretend to answer more.

Check a condition before it is written

A rule’s condition is compiled when the document is written, and a document with a bad one is refused whole. To ask about one expression first, from an editor, a CI step, or by hand, post it to the checker with the scope it will be written under:

$TOKEN is a JWT from the issuer this gateway trusts, there is no admin secret to hold. pistra login caches one and pistra admin -url presents it for you. The curl form is here because it is what a script or an SDK sends. See Sign in from the command line.

curl -sS "$ADMIN/admin/v1/config/check-condition" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"scope": "access_rules", "at": ["llm"], "expression": "args.tool == \"search\""}'

The response looks like this:

{
  "ok": false,
  "errors": [{"line": 1, "column": 1, "message": "at llm: undeclared reference to 'args'"}],
  "variables": [{"name": "key", "type": "map(string, dyn)"}, {"name": "model", "type": "string"}, ...],
  "functions": ["contains", "endsWith", "lowerAscii", "matches", "sets.contains", ...],
  "macros": ["all", "exists", "exists_one", "filter", "has", "map"]
}

args is the tool hop’s variable. With "at": ["mcp"] the same expression is "ok": true. The three scopes are access_rules (the deployment’s and a profile’s, checked at the hops at names, and with none named, against what every hop shares), guardrails (a rule’s when) and guardrails_select (a redact rule’s select). It is the same compile the write runs, the same variables, the same strings/lists/sets extensions, the same bool-output rule and cost budget, so a write accepts whatever the checker accepts. The vocabulary comes back with the verdict so that an editor completes against the gateway’s list rather than one of its own. The console’s rule fields work this way. The action is checkCondition, under the config read grant.

Run the detectors over text you choose

The suite fabricates annotations so that it can run with nothing loaded. The last question it cannot answer is whether the detectors find what the rule expects them to find. Ask the running gateway:

curl -sS "$ADMIN/admin/v1/guardrails/inspect" \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"text": "charge 4012 8888 8888 1881 to my card", "profile": "research"}'

The response looks like this:

{
  "action": "deny",
  "rule": "block-cards",
  "message": "card numbers must not be sent to the model",
  "annotations": [{"type": "pii/CREDIT_CARD", "text": "4012 8888 8888 1881", "start": 7, "end": 26, "score": 1, "validated": true, "detector": "pii"}],
  "shadow": [],
  "deferred": [],
  "errors": []
}

The text is inspected as the user turn of a chat request, by the profile’s detector selection or by the whole ensemble when no profile is named, and the decision is the one the request-side rules would have reached. It is a real run of the detectors at their real cost, a remote analyzer included, and it is not a request: nothing is forwarded, no budget moves and no guardrail.decision is recorded. The trail carries an admin.probe record with reason inspect naming the profile, never the text. The action is inspectGuardrails, under the config read grant. The console’s guardrails page has the same run under its Try it tab.

Test your rules before they ship

Write the suite

Run it

What an error verdict is telling you

Preflight and the suite

Check a condition before it is written

Run the detectors over text you choose

What an `error` verdict is telling you