Concepts

Endpoints and Routing

Stable model names, routing to deployments, weights vs failover, gateway resolution.

An endpoint is the stable public model name clients call. It’s plane-only, never syncs to a cluster, and resolves to one or more deployments with weights or a failover order. The gateway reads endpoints and their routing table, never workloads directly.

Why endpoints and deployments are separate

A deployment is cluster-specific and lives only as long as the workload is placed there. An endpoint is a contract whose name clients depend on. Merging them would give you one object that is globally unique but reconciled by N clusters at once (two writers on one row), and where the name changes whenever you redeploy.

Splitting them lets you:

Route one model name to deployments across several clusters.
Run a canary or blue-green rollout by adjusting weights between two deployments of the same model.
Rename a deployment or change its serving config without breaking client code.
Serve one private model under different endpoint names for different tenants.

The gateway needs a “public name to weighted upstreams” mapping regardless. Split, it’s a first-class object you can version and audit.

Endpoint structure

Each endpoint has:

name: Internal identifier (used in URLs, metrics labels, immutable)
slug: The model ID clients send in requests (mutable, defaults to the artifact slug)
project: Which project owns this
visibility: public or private
targets[]: List of deployments this endpoint resolves to

The slug is what clients type in {"model": "..."}. The name is what the console and metrics use. You can patch the slug without renaming the endpoint, which changes the public model ID without touching clusters or restarting anything.

Targets

An endpoint’s targets array lists the deployments it routes to. Each target has:

deployment: Which deployment (by ID or name)
weight: For weighted load balancing (default 100)
enabled: Whether to include this upstream (default true)

Weighted routing

If targets have weights, the gateway splits traffic proportionally:

{
  "targets": [
    {"deployment": "llama-70b-iad", "weight": 80},
    {"deployment": "llama-70b-fra", "weight": 20}
  ]
}

80% of requests go to iad, 20% to fra. This is how you canary a new configuration or balance across regions.

Failover routing

If you want ordered failover (try A, fall back to B only if A is down), set weights to the same value or leave them unset:

{
  "targets": [
    {"deployment": "llama-70b-primary"},
    {"deployment": "llama-70b-backup"}
  ]
}

The gateway tries them in order until it finds one with a ready replica.

Disabling a target

Set enabled: false to temporarily remove an upstream without deleting it. The gateway skips disabled targets. This is how you drain a deployment without changing its replicas: the cluster keeps serving, but new requests stop arriving.

Resolution

The gateway reads the resolved routing table via GET /v1/endpoints/resolved. This is a derived view, computed on read, never stored. For each endpoint, the control plane:

Follows each enabled target to its deployment.
Checks the deployment’s admission phase (only running deployments are included).
Checks the cluster’s phase (only active clusters contribute upstreams).
Looks up the cluster’s serving address.
Returns the list of upstreams with their weights.

An endpoint with no usable targets comes back with an empty upstream list. The gateway answers 503 no_capacity for requests to it, never 404. “No such model” and “this model has nowhere to go” are different answers.

Served model name

Each upstream in the resolution carries served_model_name: the model ID the engine behind it answers to. By default this is the artifact’s slug (e.g., moonshotai/kimi-k3), lowercase and derived from the HuggingFace source.

The gateway rewrites the request’s model field to this before forwarding, so the public name (slug) doesn’t have to match what the engine serves. Clients call gpt-3.5-turbo; the engine answers to openai/gpt-3.5-turbo; the gateway translates.

served_model_name rides the upstream, not the endpoint, because one endpoint can front two clusters running different serving configurations, and there’s no single answer to put one level up.

Output modality

Each upstream also carries output_modality: text, image, or video. This comes from the workload’s serving.output_modality and defaults to text.

The gateway uses this to refuse requests to paths the upstream doesn’t answer:

text: Answers every path except /v1/images/* and /v1/videos/*.
image: Answers /v1/images/* and /v1/chat/completions (image as image_url content part).
video: Answers /v1/videos/* only.

The check runs before an upstream is picked, so the answer doesn’t depend on which cluster would have taken the request.

Visibility

Endpoints have a visibility field:

public: Reachable from the public gateway listener. Requires the tenant’s host header.
private: Only reachable from the internal gateway listener. Not exposed outside your network.

A private endpoint is for internal callers that trust the network boundary. A request to a private endpoint via the public listener gets 404.

Multi-cluster routing

One endpoint can route to deployments in multiple clusters:

{
  "slug": "llama-3-70b",
  "targets": [
    {"deployment": "llama-70b-us-east", "weight": 50},
    {"deployment": "llama-70b-eu-central", "weight": 50}
  ]
}

The gateway picks an upstream from the resolved set according to weights, then forwards over the tunnel to the agent in that upstream’s cluster.

If one cluster’s deployment is unhealthy (no ready replicas), it drops out of the resolution and traffic shifts to the others. When it recovers, it returns and traffic rebalances.

Gateway polling

The gateway polls /v1/endpoints/resolved on a timer (default 30 seconds). It caches the table in memory and refreshes it. Requests use the cached table, so no join per request.

The table is ETagged by a revision covering every table it was built from. If nothing changed, the poll gets 304 Not Modified and costs almost nothing. When you edit an endpoint, disable a target, or change a cluster’s phase, the revision moves and the gateway gets the new table on its next poll.

A gateway that restarts during a control plane outage cannot load a routing table and refuses to start. Gateways already running keep serving from the last table they loaded.

Tenant isolation

Each endpoint belongs to one project, which belongs to one tenant. The resolved table carries the tenant_id of each endpoint’s project, so the gateway can enforce tenant boundaries.

On the public listener, the Host header identifies the tenant. The gateway only returns upstreams belonging to that tenant. A request naming another tenant’s endpoint gets 404, never 403, so a guess about what exists is never confirmed.

On the internal listener, there’s no tenant check. Any caller can reach any endpoint.

Endpoint creation

You typically create an endpoint alongside the first deployment, so the simple path stays one call. POST /v1/workloads with create_endpoint: true creates both. The endpoint’s slug defaults to the artifact’s slug unless you override it.

You can also create endpoints standalone via POST /v1/endpoints and add targets later. This is how you set up weighted routing or failover before deploying anything.

Renaming and aliasing

The endpoint’s slug is patchable. Changing it updates the public model ID clients send, with no cluster changes. A patch to the slug costs one gateway poll (to refresh the routing table) and nothing else.

If you want an alias (two names for the same deployments), create a second endpoint with a different slug pointing at the same targets.

The endpoint’s name is immutable. It’s in metrics labels and URLs. Renaming it would change what every referring metric series says.