Skip to content
pistra

Add a custom provider

Add a custom provider when your endpoint has no catalog preset. Set its name, dialect, and URL. Use this for a self-hosted server, an internal proxy, or a second instance of a known provider.

The whole thing

Section titled “The whole thing”

Add this configuration:

providers:
  - name: vllm
    dialect: openai
    base_url: http://vllm.inference.svc.cluster.local:8000
    api_key: ${secret:VLLM_API_KEY}
    models: ["llama-*", "qwen-*"]

Set these fields:

  • name identifies the provider in routes, rules, budgets, logs, and metrics. It is required when you do not use a preset.
  • dialect names the protocol the endpoint speaks, openai or anthropic. It decides whether a request is forwarded byte-for-byte (the client spoke the same dialect) or translated on the way (it did not). A server that serves /v1/chat/completions is openai. One that serves /v1/messages is anthropic. There is no third value, and no way to declare one from config. A dialect ships with a translator, a probe, and a coverage fixture.
  • base_url is the root the request path is appended to. Any absolute URL, including http. Catalog endpoints must use https. Your deployment endpoint does not have to.
  • api_key (or credential) is sent upstream in the header the dialect expects, in place of the caller’s own credential. It is required whenever auth is set even if the server ignores it. Without one, the gateway would forward the caller’s credential upstream. If you omit auth, omit this field to forward the caller’s credential unchanged.
  • models gates what routes here, exact names or trailing-* prefixes. The gateway also uses this list for GET /v1/models.

Send a request and watch it arrive:

Terminal window
$ curl -s localhost:8484/v1/chat/completions \
    -H "Authorization: Bearer $VIRTUAL_KEY" \
    -d '{"model":"llama-3.1-8b","messages":[{"role":"user","content":"hi"}]}'

Check GET /admin/v1/backends to inspect the resolved URL and credential. Custom providers have one native channel named default. Requests report it as pistra.channel.

What a custom provider is, to the router

Section titled “What a custom provider is, to the router”

A preset includes catalog facts such as endpoint quirks and channel claims. A custom provider has one native channel in its declared dialect.

The gateway translates requests from the other dialect. Apply on_fidelity_loss and require_caps as you would for any route. A custom provider cannot declare a provider_compat channel. Add an overlay when you need to publish that claim with a source and verification date.

Preset with an override, or custom?

Section titled “Preset with an override, or custom?”

Both point a known provider somewhere else. They keep different things.

you want write
OpenAI through a corporate proxy, Anthropic via a VPC endpoint, Bedrock in a region preset: plus base_url:, the dialect, quirks and channel claims stay
the same provider twice, under two names or two keys name: plus preset: twice, an unnamed provider is addressed by its preset, so the second needs a name
an endpoint that speaks the dialect but is not that provider name:, dialect:, base_url:, no preset
a fleet of identical servers you reach directly pool: instead of base_url:, see schedule a pool

The line is whether the catalog’s facts about the provider are true of your endpoint. A proxy in front of api.openai.com still strips the fields OpenAI rejects. A vLLM serving the same paths does not want them stripped. dialect cannot be set alongside preset for the same reason. It is a fact the preset owns, not a default it suggests.

Three things that bite

Section titled “Three things that bite”
  • A server that rejects unknown fields. pistra adds stream_options.include_usage to streamed OpenAI-dialect requests so usage can be metered, and removes the chunk it produces. A server that returns 400 on a field it does not know needs stream_usage: false, and its streamed requests then settle against the reservation estimate.
  • A path prefix the server does not serve. The incoming path is appended to base_url as it is. A server that serves /chat/completions without the /v1 has no config-side fix. It needs an overlay entry with strip_path_prefix, or a base_url whose server adds the prefix back.
  • region without a preset. It fills a placeholder only a region-parameterized preset has. On a custom provider it is refused unless the credential is aws, where it is the SigV4 signing region.

From the console or the API

Section titled “From the console or the API”

In the console, New provider offers Preset and Custom as peers: a preset from the deployed catalog, or a name, a dialect and a URL. Either way the object lands in the source you chose and is validated as the whole deployment on write.

Through the API it is one object write, naming the source only because the provider does not exist yet:

Terminal window
$ curl -s -X PUT 'localhost:8485/admin/v1/config/providers/vllm?source=team-a' \
    -H "Authorization: Bearer $ADMIN" -H 'If-None-Match: *' \
    -H 'Content-Type: application/yaml' \
    --data-binary $'name: vllm\ndialect: openai\nbase_url: http://vllm:8000\napi_key: ${secret:VLLM_API_KEY}\nmodels: ["llama-*"]\n'

See manage the configuration through the API for the precondition and what a refused write looks like.

When to promote it

Section titled “When to promote it”

A custom provider is a fact about your deployment. When it is a fact about the world, move it to an overlay. That covers a public provider with a stable endpoint, a documented compat surface, or quirks other deployments would hit. In an overlay it gains provenance, a validator, and a place in /admin/v1/catalog/report. Nothing changes in the providers: entry except that it now says preset: instead of dialect: and base_url:.

Section titled “Related”