Design

Data Plane: Gateway and Tunnel

Gateway architecture, tunnel protocol, agent dial-out, role reversal, tenant isolation.

The gateway, tunnel, and agent are how inference requests reach models. None are part of the control plane, and while running, none need it to be up. The exceptions are at startup: the gateway needs the plane to load its first routing table, and an agent that redials needs the gateway to verify its token (which the gateway can answer from cache for up to an hour without the plane).

Gateway architecture

libs/gateway takes an OpenAI-compatible request, reads model from the body, and forwards it to a cluster currently serving that model. It is one hop. The plane derives the routing table; the gateway keeps a copy in memory and refreshes it on a timer, so nothing is joined per request.

Failure modes

The plane is down: the table already loaded keeps serving. What it’s built from changes on the order of days, and a control-plane outage must not stop traffic to models running fine.
It boots with no table: it refuses to start. Coming up empty would answer everything with model_not_found, which reads as an outage of the models instead of the misconfiguration it is. The table is held in memory only, so this includes a gateway restarted while the plane is down: a restart, eviction, or scale-up during a plane outage leaves that replica out until the plane returns. Persisting the last table is on the roadmap.
An endpoint has no upstreams: 503 no_capacity, never 404. “No such model” and “nowhere to send it” are different answers, and a caller acts differently on each.

What the gateway does not do

The gateway does not meter, bill, or pick replicas. Whoever sells inference already has a place where metering happens, and the serving runtime inside a cluster routes on queue depth and KV-cache residency—state that does not and should not leave the cluster.

It does identify tenants. Which listener accepted a connection is the trust primitive: the internal listener has no ingress, so reaching it is the proof; on the public listener the Host header names a tenant, and a tenant that declares auth: api_key must present a key belonging to it. Another tenant’s model reads as 404, never 403.

Gateway as failure domain

The gateway is one failure domain. It is a single deployment, wherever it runs, so if every replica is down every model is unreachable through Wingsmith, in every region. Run several replicas across zones; regional gateways are not designed yet. Its internal listener has no authentication and must be kept off any ingress and behind a NetworkPolicy.

Because it sees every request, including those that never reach a cluster, the gateway is the one complete source of a failure rate. It counts each request under a closed set of outcomes, keeping “caller over its own limit” and “this hop shedding load” apart even though both are 429.

Tunnel protocol

Nothing dials into a GPU cluster, not even the gateway. The agent (libs/agent) dials out to the gateway, authenticates with the cluster’s enrollment token, and then the roles reverse: the agent runs the HTTP/2 server on that connection and the gateway runs the HTTP/2 client. Both ends of that wire live in libs/packages/tunnel, a module of its own because neither side owns it.

Why role reversal

Wingsmith carries its own tunnel, built on HTTP/2 with the roles reversed. The agent dials the gateway over TLS, presents its enrollment token in a single length-prefixed handshake (HGRTUN01), and then runs the HTTP/2 server on that connection while the gateway runs the HTTP/2 client. The gateway therefore holds an ordinary http.RoundTripper per cluster.

The alternative options were:

Option	Why not
frp	A separate `frps` server in the path. Routing is by its own proxy configuration, not per request from Go. Auth is its own token, not the enrollment token.
chisel	Hands the gateway a local port per remote, so the gateway would proxy to `localhost:N` and lose which cluster and which tunnel a request is on.
cloudflared / ngrok	Ties a self-hosted data path to a vendor account and edge. The tunnel terminates at the vendor, not at the gateway, and every prompt crosses a third party.
Konnectivity	The closest in shape. Built around the Kubernetes API server’s dial semantics; using it means carrying its server and gRPC framing for an HTTP-only path.
Inbound load balancer per cluster	Rejected by the architecture: it’s the push model’s reachability and credential problem moved to the data plane.

What the tunnel provides

An http.RoundTripper per cluster. The gateway is an httputil.ReverseProxy. It needs to pick a cluster per request and hand the request to something that speaks HTTP to that cluster, with streaming responses, per-request cancellation, and a response-header timeout.
Retry of unprocessed requests on a sibling connection. An agent rollout drains its tunnels with a GOAWAY. Requests the agent never started must move to another live tunnel to the same cluster, not fail as 502s. A request that already streamed bytes to the caller must not be retried.
A fast answer when a cluster is full, instead of queueing behind a stream limit, and errors classified so that tunnel churn is not recorded as a cluster outage.
Identity is the cluster’s enrollment token. The token the operator already holds, verified by the plane, decides which cluster a tunnel belongs to. A second credential system for the data path would be a second thing to mint, rotate, and revoke per cluster.
Coverage of every gateway replica from one address. The tunnel pool is in-process, so each gateway replica can only serve clusters it holds a tunnel to. An agent given one address has to find all replicas, without the cluster’s values listing them.
No extra process or vendor in the request path. The gateway is already the one Hangar process in the path. A separate tunnel server or a hosted edge adds a hop, a failure domain, and, for a hosted edge, an account and a third party that sees every prompt.

Replica discovery

The agent is given one gateway address and finds every gateway replica behind it: each accepted handshake names the replica that took the dial and how many there are, so the agent keeps redialling until it covers them all. A cluster is never configured with the shape of the control plane, and the gateway can scale without an edit reaching any cluster.

The pool implements http2.ClientConnPool, so http2.Transport’s own retry runs and the pool hands it a different live tunnel. Connection resets that x/net does not retry are retried by the pool only before any byte has reached the caller.

The response-header timeout is the upstream one, armed at request write and disarmed at the first header. A full cluster returns ErrSaturated at once instead of blocking for a stream slot.

Separation from control plane

The agent and the operator are separate processes with separate connections. If a request needed the operator’s sync connection, control-plane uptime would be the inference SLA again.

Routing inside the cluster

Inside the cluster, the operator renders a frontend for each model as part of that model’s deployment, instead of one shared frontend for the cluster. The gateway forwards the deployment id it resolved as X-Hangar-Deployment-Id, and the agent maps it to the frontend Service labeled with that id.

One frontend per model

The operator renders a Frontend component into the graph it already renders per deployment. One frontend per model. It is no longer cluster-setup’s singleton, and a model’s runtime is no longer anybody else’s constraint.

The gateway forwards the deployment id it already resolved, as X-Hangar-Deployment-Id. The constant lives in libs/packages/tunnel, because neither end owns the wire. The gateway sets it unconditionally and deletes it when there is none: ReverseProxy clones inbound headers verbatim, so anything less lets a caller pick which model’s frontend answers them.

The agent maps that id to a Service by the hangar.dev/deployment-id label. Dynamo copies a component’s pod-template labels onto the Service it derives. A frontend Service with no such label is the cluster’s shared frontend and is the fallback.

Why not `/v1/models` polling

Two deployments of one model on one cluster (a canary, a ramp) advertise the same string, so a probe cannot tell them apart and would silently collapse a weighted rollout the gateway had already split, while the metrics kept attributing traffic to the deployment the gateway picked. It is also the exact signal that broke: a frontend whose model list is empty is the thing to detect, not the thing to route on.

Cache isolation

Cache salt per tenant for KV isolation, media salt for multimodal. The gateway writes tenant-specific salts into request headers that the frontend reads, so KV cache entries are scoped by tenant and cannot be shared across them.

Threat model

Assets: Inference traffic (prompts and completions), each cluster’s enrollment token, and the mapping from a tunnel to a cluster.

Boundaries: The tunnel listener is reachable from wherever clusters run, usually the internet. The agent runs inside a cluster the operator of the installation controls.

Threat	Control	Residual risk
Eavesdropping or tampering between agent and gateway	TLS on the tunnel listener (`-tunnel-cert`/`-tunnel-key`), or TLS terminated by a TCP-passthrough load balancer in front; the agent verifies the gateway’s certificate	A deployment that terminates TLS in front of the listener and forwards plaintext over an untrusted network; `HANGAR_TUNNEL_INSECURE_SKIP_VERIFY` is for local development only
Unauthenticated dials exhausting the gateway	10 s handshake deadline, 8 KiB handshake cap, rejected dials counted by reason	Volumetric attacks on the listener are left to the network edge
A dial claiming to be another cluster	Identity is whatever the plane resolves the token to (`POST /v1/clusters/tunnel-auth`, called with the gateway’s service token); a claimed cluster id that disagrees is rejected	None beyond token theft
A stolen enrollment token	Tokens are per cluster, so a stolen one impersonates that cluster only; rotating it in the plane refuses it on the next verification	Verifications are cached for 5 minutes, and for up to an hour while the plane is unreachable, so a revoked token can open new tunnels for that long. Tunnels already open stay open until they drop.
A compromised cluster	It can only attach tunnels for its own cluster, so it only receives traffic routed to models placed on it. HTTP/2 stream limits, flow-control windows, and the response-header timeout bound what it can make the gateway hold	It sees the prompts sent to its own models, as any serving cluster does, and can answer them wrongly
Routing headers forged by a caller	The gateway overwrites `X-Hangar-Deployment-Id` on every request, and the agent trusts it only from a tunnel	None known
A compromised gateway	Out of scope for the tunnel: the gateway is in the request path by design and sees all traffic	Run it as trusted infrastructure

Tunable limits

The tunnel module exposes several limits that control resource use and failure behavior:

MaxConcurrentStreams: The maximum number of concurrent HTTP/2 streams per tunnel. When reached, the pool returns ErrSaturated instead of blocking.
ResponseHeaderTimeout: How long to wait for the first response header after writing the request. The upstream http2.Transport timer.
MaxHandshakesPerSource: Per source address rate limit on handshake attempts.
SourceFailures: How many consecutive failures from a source before it’s temporarily blocked.
ReverifyInterval: How often to re-verify an enrollment token with the plane.
CacheTTL: How long to cache a verified token.
CacheGrace: How long to continue accepting a cached token when the plane is unreachable.

Per-source limits key on the dial’s source address. They assume the listener sees the agent’s own address. Behind a load balancer or Service that rewrites it, every agent shares a few node addresses and the limits would throttle them together; such a deployment should preserve the client address or set both limits negative.