Calling the gateway

OpenAI-compatible inference API: paths, listeners, tenant keys, streaming, limits, errors and troubleshooting.

The gateway takes an OpenAI-compatible request, reads model from the body, and forwards it to a cluster that serves that model. It is a separate process (hangar-gateway) with its own address, not part of the control plane API. For how it works inside, see Data plane.

Supported paths

Only these paths are forwarded. All are POST except GET /v1/models, which the gateway answers itself.

Any other path or method is 404 unsupported_path, answered before the body is read. That includes engine routes such as /tokenize, /health and /metrics. There is no job store for video, so GET, DELETE and listing under /v1/videos are also 404 unsupported_path.

Output modality

Each endpoint generates text, image or video, and the gateway refuses paths the endpoint cannot answer with 404 unsupported_path.

GET /v1/models returns the OpenAI list shape with an extra output_modality field on each model:

{
  "object": "list",
  "data": [
    { "id": "llama-3-8b", "object": "model", "owned_by": "system", "output_modality": "text" }
  ]
}

id is the endpoint’s public name. It is the value to send as model. The gateway rewrites it to the name the engine expects before forwarding. owned_by is always system.

Listeners and authentication

The gateway runs two listeners over one routing table. Which one accepted the connection decides how you authenticate and which models you can reach. See Tenants and projects for the ingress and auth settings.

On the public listener:

• A model that belongs to another tenant, or is private, reads as 404 model_not_found, never 403.
• A tenant with auth: none is refused with 401 unauthorized unless the gateway runs with HANGAR_GATEWAY_PUBLIC_ALLOW_NO_AUTH=true.
• A Host that matches no public tenant is 404 unknown_host. A tenant created moments ago can get this until the gateway’s next resync (30 seconds by default).

A disabled tenant gets 403 tenant_disabled on both listeners.

Tenant keys

A tenant key (hgrk1...) belongs to one tenant and is minted under Manage tenants. Send it as a bearer token:

Authorization: Bearer hgrk1...

The gateway does not read X-Api-Key. It strips that header, along with Authorization, Cookie and Proxy-Authorization, before forwarding, so no cluster sees your key. A missing, unknown, revoked or other-tenant key is 401 unauthorized. Keys are not accepted by the control plane API; see Authentication.

Make a request

The public listener sits behind a wildcard host, one subdomain per tenant. The examples use acme.gateway.example.com for the tenant acme.

curl https://acme.gateway.example.com/v1/chat/completions \
  -H "Authorization: Bearer $HGR_TENANT_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3-8b",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "max_tokens": 100
  }'

On the internal listener, drop the key and call the gateway address directly:

curl http://gateway.internal:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model": "text-embedding-ada-002", "input": "The quick brown fox"}'

Request and response bodies are the OpenAI formats. The gateway returns the engine’s response unchanged, except for errors.

Streaming

Set "stream": true and read server-sent events. The stream ends with data: [DONE]. The gateway forwards bytes as they arrive and does not buffer or parse the stream.

curl -N https://acme.gateway.example.com/v1/chat/completions \
  -H "Authorization: Bearer $HGR_TENANT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-3-8b", "stream": true, "messages": [{"role": "user", "content": "Hello"}]}'

A stream is never cut for being long, only if you stop reading: one blocked write to you is limited by HANGAR_GATEWAY_WRITE_IDLE_TIMEOUT (default 2 minutes).

Other request rules

• stop accepts at most four sequences on /v1/chat/completions and /v1/completions, otherwise 400 too_many_stop_sequences.
• Routing hints in the body (such as nvext worker pins) and steering headers (x-dynamo-*, x-worker-instance-id, x-tenant-id and similar) are removed before forwarding. You cannot pick a worker.
• Image and video responses carry no token usage. The gateway records output counts and size instead.

Limits

Request limits

All variables are documented in Gateway configuration.

Rate limits

Limits apply per tenant, not per key, and per gateway replica. The fleet-wide figure is the per-replica value times the replica count.

A tenant over either limit gets 429 rate_limited with a Retry-After header. On the public listener the check runs before the body is read. On the internal listener it runs after the model is resolved, because that is when the tenant is known.

These are protective limits. The serving quota an endpoint declares is not enforced by the gateway yet.

Video input

Video input is bounded in decoded bytes, not frames, so a short high-resolution clip and a long low-resolution one are held to the same ceiling. Where each bound is enforced:

• Gateway: a media_io_kwargs field in the request is silently dropped, so a caller cannot override the sampling rate set on the workload. A data: URL clip is bounded only by the body size limit above.
• Workload: serving.max_videos_per_request caps videos per request, and the bounds assume 1. A request over the cap is 400.
• Worker: the fetch size of a remote clip and its decoded size are bounded in the worker’s video loader, which refuses with 400 naming the clip’s duration and resolution. The gateway cannot see inside a remote URL, so it does not enforce these.

See Decisions for the reasoning.

Remote media URLs

Image, video and audio inputs can be given by URL. The engine fetches them from inside the serving cluster, so the gateway checks every remote media URL first and refuses one it does not allow with 400 media_url_not_allowed. data: URLs carry their bytes and always pass.

The check is by name, so a public name that resolves to a private address gets through. It does not replace the engine’s own allow-list or an egress NetworkPolicy on the worker namespace. Use all three where tenants can send media URLs. Multipart bodies carry media as bytes and are not checked.

Prefix-cache isolation

Every text generation request (/v1/chat/completions, /v1/completions, /v1/responses, /v1/messages) is salted with its tenant, so two tenants on one deployment never share KV cache blocks. You do nothing to get this.

On the internal listener, X-Hangar-Cache-Scope narrows the salt further. An operator serving many customers through one tenant can name the customer there:

curl http://gateway.internal:8080/v1/chat/completions \
  -H "X-Hangar-Cache-Scope: customer-42" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-3-8b", "messages": [{"role": "user", "content": "Hello"}]}'

The header is ignored on the public listener and never reaches the cluster. A request that carries media is salted with a hash of its prompt and media, so a multi-turn conversation with media does not reuse the prefix cache across turns. Text-only requests are unaffected. See Data plane.

Responses and errors

X-Request-Id

Every response carries X-Request-Id. If you send one of at most 128 characters from [A-Za-z0-9._:-], the gateway keeps it. Otherwise it mints a UUIDv4. The same ID is in the gateway’s access log line and is forwarded to the cluster, so quote it when you report a problem.

Error format

Errors use the OpenAI envelope. type follows the status and code is what to branch on:

{
  "error": {
    "message": "The model 'nope' does not exist.",
    "type": "invalid_request_error",
    "code": "model_not_found"
  }
}

Engine errors are rewritten. A 4xx from the engine keeps its class with a fixed or sanitised message (invalid_request for most; 422 becomes 400). An engine 401, 403 or 404 becomes 502 upstream_error, because those mean a misconfiguration on the gateway side. A 5xx becomes 502 upstream_error with a fixed message, an engine 503 becomes 503 no_capacity, and an engine 504 becomes 502 upstream_unavailable. The engine’s own text is never quoted on a 5xx, since it can carry paths or addresses.

Troubleshooting

What the gateway keeps

It keeps no request or response content. The access line has the request ID, method, path (never the query), status, outcome, timings, token counts, the tenant and a 12-character key-hash prefix. It never logs a body, a header value or a model’s answer.