Deploy the gateway
Run the gateway: what it needs from the plane, listeners, replica count, tunnel TLS, readiness and plane-outage behaviour.
The gateway is the only Wingsmith process in the inference request path. It reads the model from an OpenAI-compatible request and forwards it to a cluster that serves that model. It also terminates the tunnels cluster agents dial in on. For how the pieces fit together, see Data plane.
It runs as one StatefulSet, hangar-gateway, with stable pod names. The image is gateway. Every setting is in Gateway configuration.
What it needs from the plane
The gateway holds no state of its own. It loads a routing table and a tenant table from the plane’s API and refreshes both every 30 seconds (-resync).
| Setting | What |
|---|---|
HANGAR_PLANE_URL | the plane api’s base URL as the gateway reaches it, for example the api’s in-cluster Service |
HANGAR_GATEWAY_SERVICE_TOKEN | the same value as on the plane’s api |
Both are required. The service token is accepted on three plane routes only: the resolved endpoints, the gateway tenant feed, and tunnel authentication. It cannot be used for anything else.
The first load of each table must succeed. If the plane is unreachable or rejects the token at boot, the gateway logs the error and exits. See Plane outages.
Listeners
| Listener | Default | Who connects | Expose it as |
|---|---|---|---|
| internal | :8080 | your own services that call models | ClusterIP only, behind a NetworkPolicy |
| tunnel | :8443 | every cluster’s agent | L4 passthrough |
| public | off | tenants calling directly | HTTPS ingress on a wildcard host |
| metrics | :9090 | your Prometheus | cluster network only |
The internal listener authenticates nobody. Reaching it is the proof of identity, and a caller can reach every enabled tenant’s models. Kubernetes has no network boundary by default, so apply a NetworkPolicy that admits only your own inference edge. The repository’s libs/gateway/deploy/networkpolicy.yaml is an example that also admits the ingress controller to the public listener, agents to the tunnel port and monitoring to :9090. Adapt its selectors. Never give the internal listener an ingress.
The public listener is on only when HANGAR_GATEWAY_PUBLIC_LISTEN is set. It identifies the tenant from the Host header, one subdomain per tenant under a wildcard host, and checks an API key when the tenant requires one. Only tenants whose ingress is public are served, and only their public endpoints. Tenants and their settings are in Manage tenants.
The tunnel listener is HTTP/2 after a custom handshake, so an HTTP ingress cannot proxy it. Use a TCP route or an L4 load balancer.
Replicas and tunnel coverage
Agents dial out, and every gateway replica needs a tunnel from every cluster it routes to. A replica with no tunnel to a cluster cannot serve that cluster’s models. An agent is configured with one address. Each accepted handshake reports which replica took the dial and how many replicas exist, and the agent redials that one address until it has reached all of them.
The gateway has no Kubernetes client, so you tell it the count:
| Setting | Value |
|---|---|
HANGAR_GATEWAY_REPLICAS | must equal the StatefulSet’s replicas |
HANGAR_GATEWAY_ID | this replica’s identity. Defaults to the hostname, which is the pod name in a StatefulSet |
If HANGAR_GATEWAY_REPLICAS is lower than the real count, agents stop dialling early. The replicas they missed hold no tunnel for those clusters and answer 503 no_capacity for their models while reporting healthy. If it is higher, agents make some extra dials and nothing else breaks. Unset, every replica reports one, which is only correct for a single replica, and the gateway logs a warning at startup.
Change the StatefulSet’s replicas and HANGAR_GATEWAY_REPLICAS in the same rollout. The pods restart, every agent redials and each reads the new count. Keeping the two numbers in step by hand is a known limitation.
Run more than one replica with a PodDisruptionBudget, and spread them across zones. If every replica is down, every model is unreachable through Wingsmith.
TLS
The chart always configures agents to dial the tunnel over TLS. Its agent.gateway.address takes no scheme and assumes TLS.
| Setting | What |
|---|---|
HANGAR_GATEWAY_TUNNEL_TLS_CERT, HANGAR_GATEWAY_TUNNEL_TLS_KEY | PEM pair the gateway terminates tunnel TLS with. With neither set the listener is plaintext, so something in front must terminate TLS |
Give the cert a name that matches HANGAR_TUNNEL_GATEWAY on the plane. If the certificate comes from a private CA, set agent.gateway.caBundle in each cluster’s chart values. If the name differs from the dial address, set agent.gateway.serverName. See Serving agent.
Terminate TLS for the public listener at your ingress.
Credentials for private serving addresses
Some clusters sit behind a credential on their serving address. The plane stores only a reference to the credential, never the secret. Mount a JSON file that maps each credential id to the headers to send, and point HANGAR_GATEWAY_CREDENTIALS at it. Leave it unset when no serving address needs one.
Readiness
| Probe | Behaviour |
|---|---|
/healthz | liveness. Always 200. Do not make it fail for a missing tunnel, because restarting does not fix one |
/readyz | readiness. Fails while the replica is draining, or when it can reach none of the clusters in its routing table. An empty routing table counts as ready |
A replica missing one cluster’s tunnel stays in the Service and answers 503 for that cluster’s models. The metric hangar_gateway_tunnel_coverage shows which replica lacks which cluster. See Metrics.
The tunnel listener is not gated on readiness, because a replica becomes ready by holding a tunnel. If agents could only dial ready replicas, none could ever become ready. Put the tunnel behind a headless Service that publishes not-ready addresses.
Shutdown
On SIGTERM a replica reports not-ready, keeps accepting briefly while load balancers drop it, drains the request listeners, then drains the tunnel listener. The phases are cut from HANGAR_GATEWAY_SHUTDOWN_GRACE, which must equal the pod’s terminationGracePeriodSeconds (30 seconds by default). To carry longer generations through a rollout, raise both together.
When the plane is down
A running gateway keeps serving from the tables it last loaded, and still accepts agent reconnects for tokens it verified within the last 15 minutes (HANGAR_GATEWAY_TUNNEL_CACHE_GRACE). A gateway that starts while the plane is down does not come up, so a restart, eviction or scale-up during an outage leaves that replica out. Do not roll the gateway while the plane is unavailable. See Plane outages.
Check it
Probe
GET /healthz and GET /readyz on the internal listener return 200.
Enrol a cluster
After enrolling a cluster with the agent on, hangar_gateway_tunnel_coverage shows a tunnel from that cluster on every replica.
Send a request
Call the internal listener with a model that is served. See Gateway API.