Plane outages
What keeps working, what stops, and what you see when the Wingsmith control plane is unreachable.
Inference does not depend on the plane being up. Clusters keep running what they were last told to run, and a running gateway keeps serving from the routing table it last loaded. What you lose is the ability to change anything, and the ability to start new gateway replicas.
What keeps working
| Part | Behavior during the outage |
|---|---|
| Inference requests | The gateway serves from its cached routing table. Agents forward to frontends already running |
| Running deployments | The operator reconciles from its local SyncedResource objects, including across an operator restart |
| Operator | Redials the hub with backoff (1 s to 30 s, full jitter) and resumes from its durable checkpoint when the hub returns |
| Agent tunnels | Open tunnels stay open. A reconnecting agent is admitted if the gateway still holds a verified token for it (see below) |
What stops
| Part | Behavior |
|---|---|
| Gateway start | A gateway that starts during the outage has no routing table and refuses to start. A restart, eviction or scale-up leaves that replica out |
| New declarations | Changes made earlier through the API do not reach clusters. A new deployment has no cluster status until the hub can stream it |
| Routing table refresh | The gateway serves what it last loaded. Deployments added or removed during the outage are not reflected |
| Observations | The operator queues observed state and events in memory, bounded by HANGAR_SEND_QUEUE_SIZE (default 1024). Past the bound, events are dropped and counted, and a newer observation replaces the queued one for the same subject. A restart loses the queue |
Console, hgr, API | Connection errors |
| Registration | New clusters cannot register and you cannot mint enrolment tokens |
Because the gateway refuses to start without a table, run more than one replica spread across zones with a PodDisruptionBudget, and do not roll the gateway while the plane is down.
Agent reconnects
The gateway verifies an agent’s token with the plane. A verified token is reused for HANGAR_GATEWAY_TUNNEL_CACHE_TTL (default 1 minute) and survives an unreachable plane for HANGAR_GATEWAY_TUNNEL_CACHE_GRACE (default 15 minutes). After the grace period, a dial for that token fails until the plane is back. Open tunnels are not closed by the outage, only reconnects are affected, so a cluster that loses its tunnels in a long outage cannot re-establish them until the plane returns.
What you see
Gateway. hangar_gateway_routing_table_age_seconds climbs while hangar_gateway_routing_table_refresh_success is 0 and the error count in hangar_gateway_routing_table_refreshes_total grows. Together they are the plane outage as seen from the data plane. /readyz stays ready as long as the replica holds a table and can reach at least one cluster in it. A replica missing one cluster’s tunnel stays in the Service and answers 503 no_capacity for that cluster’s models.
Operator. /healthz returns 200 and /readyz returns 503 while the hub is unreachable. The pod is alive but not ready, Kubernetes does not restart it, and the cluster keeps running declared workloads. The operator logs each reconnect attempt.
Plane. Check /health on the api and hub pods, and Postgres. The hub’s session state lives in memory per pod, so a hub restart gives every cluster a full resync on reconnect instead of a resume.
Recovery
Nothing needs doing by hand.
- The hub accepts connections and operators reconnect. Each either resumes or runs a full resync.
- Agents redial and the gateway verifies tokens.
- The gateway refreshes its routing table. The age gauge drops and
refresh_successreturns to 1. - Queued observations flow up. New declarations flow down.
Check that every cluster shows as connected in the console before declaring the incident over. A cluster that stays not ready needs its operator’s logs.
Scenarios
| Scenario | Result |
|---|---|
| Plane down, gateway replica restarted | The replica does not start. Other replicas keep serving. With a single replica, inference is down until the plane returns |
| Plane down, operator restarted | The operator starts, reports not ready, reconciles from local state. Workloads stay up |
| Plane down longer than the token grace | Existing tunnels stay open. Agents that lose a tunnel cannot redial |
| Deployment created just before the outage | It has no cluster status until the hub streams it |
Plane maintenance, a Postgres restart or a hub redeploy therefore does not drop inference traffic. See Upgrades for rolling order, and Monitoring for the series above.