How it works
The mental model: intent versus truth, outbound connections, control plane versus data plane, and what keeps working when the plane is down.
Four ideas explain most of Wingsmith’s behaviour. Each links to the page that covers it in depth.
Intent and truth
The control plane stores intent. The cluster stores truth.
Intent is what you asked to run: a workload, its serving configuration, scale bounds, priority. Truth is what is running: which replicas are up, on which devices, how healthy. The plane keeps intent in Postgres. Each cluster keeps truth locally and reports it up, and the plane copies it into the same database so the console and placement never fan out across clusters to answer a question.
A cluster-resident operator closes the gap. It receives desired state from the plane, renders it into Kubernetes objects (today, NVIDIA Dynamo DynamoGraphDeployment resources), and reports what it observes. Desired state syncs down. Observed state syncs up.
Because the two are separate records, a cluster keeps serving when the plane is down, and the plane can adopt a workload that was already running without restarting it.
Nothing dials into a cluster
Every cluster runs two Wingsmith processes, and both connect outward.
| Process | Connects to | Over | Carries |
|---|---|---|---|
| Operator | Hub (part of the control plane) | WebSocket | Desired state down, observed state up |
| Agent | Gateway | HTTP/2 tunnel | Inference requests into the cluster |
The plane holds no cluster credentials. An operator authenticates with a per-cluster enrolment token that allows it to sync that cluster’s state and nothing else. Outbound connections also pass most firewalls without per-cluster inbound rules, and a bad plane release cannot push a change to every cluster at once. Details are in Architecture overview and Networking.
Control plane and data plane
Inference traffic never passes through the control plane.
The control plane is the API, the hub and the reactor, over one Postgres database. The data plane is the gateway, the tunnels and the agents.
The gateway reads the model field of an OpenAI-compatible request, looks it up in its cached routing table, and forwards the request through the tunnel to the agent in a cluster that serves that model. The plane derives the routing table and the gateway refreshes it on a timer. See Data plane.
Clusters are chosen globally, devices locally
The plane picks clusters. The cluster’s scheduler picks GPUs.
Placement across clusters uses capacity, device class, region, residency, quota and policy, and it changes rarely. Placing a replica on specific GPUs depends on live topology and changes constantly, so it stays with the scheduler in the cluster. See Quotas and placement.
The plane also never sends Kubernetes manifests. It sends Wingsmith’s own objects, such as a deployment, and the operator expands them. All knowledge of Kubernetes stays on the cluster side. The objects are described in Object model.
When the plane is down
| Component | Keeps working | Stops working |
|---|---|---|
| Gateway | Serves from the last routing table it loaded | Cannot load a table at startup, so a gateway restarted during an outage does not start |
| Operator | Reconciles the last desired state it synced and queues observed state in a bounded buffer for when the plane returns | Receives no new desired state |
| Agent | Proxies inference requests over its open tunnels. A redial is accepted from the gateway’s token cache for a while (15 minutes by default) | Cannot reconnect once that cache has expired, because the gateway cannot verify its token |
| Console | Fixtures mode, with example data | Reads and writes of real state |
Models keep serving and clusters keep healing replicas. What stops is change: new workloads, scale edits, drains, cordons and routing table refreshes. Recovery steps are in Plane outages.