Providers and models

Getting traffic to something that answers it. Most of this section is shorter than it looks, because most endpoints need three facts and nothing else.

Start by assuming it just works

A provider needs a name, a dialect and a URL. Anything that speaks OpenAI’s or Anthropic’s wire format is reachable that way, your own vLLM, a regional endpoint, a vendor nobody has heard of, with no preset and no catalog entry. → Add a custom provider

Reach for the catalog only when pricing or capabilities are wrong

pistra ships a catalog of models with their prices and what each one can do, and budgets are denominated against it. When it is missing a model you are already using, or wrong about one, an overlay corrects it at load time rather than waiting for a release. → Add a provider with a catalog overlay

The distinction is worth holding on to: a provider is where requests go, and the catalog is what a request is known to cost. You can add the first without touching the second.

Then check that the numbers are the provider’s

The gateway never counts tokens itself. It reads the provider’s own counts off the response, which is why metering works at all. Reconciliation is possible rather than circular, because the ledger can be compared against the bill, and the drift has a metric. → Audit your metering against the provider

Related

Channels and fidelity covers which provider serves a given request, and why a route is a (provider, model, dialect, capability) tuple rather than a provider name.