Design

Architecture overview

Topology, design principles, processes and known limitations of Wingsmith, for people evaluating or contributing to it.

Wingsmith is a control plane for serving models across several Kubernetes GPU clusters. It handles cluster enrolment, device inventory, placement, model lifecycle, serving configuration, quota and operations, plus a gateway that routes inference requests. Inside each cluster, models are served by rendering them into NVIDIA Dynamo graphs.

It is pre-release: there is no tagged release and no published image or chart, so installation means building from source. Planned work is on the roadmap.

For the objects the system manages, see the object model. For how Wingsmith relates to Dynamo, llm-d, KServe and others, see the comparison.

Scope

Standalone

Wingsmith owns its database, deployment and vocabulary. A system that sells or meters inference talks to it through the API.

Inference serving only

Everything rendered today is a long-running model server. The object model leaves room for batch and gang-scheduled work, but no renderer exists for them.

NVIDIA Dynamo in the cluster

The plane never sends Kubernetes manifests, so a second renderer is possible. The operator renders only Dynamo graphs today, and clusters run the Dynamo platform that the cluster-setup chart installs.

You own every cluster

No third-party clusters and no air-gapped installs. This simplifies one object, Pool, and is easy to read too much into.

By design, Wingsmith is not:

A device scheduler. The scheduler inside each cluster picks cards. Wingsmith picks clusters.
A Kubernetes dashboard. Showing CRD trees and asking operators to reconstruct intent is what it replaces.
A training or batch platform. It does not run jobs to completion and does not preempt.
In the request path. Inference traffic goes through a separate data plane that keeps serving when the control plane is down. A gateway that restarts during a plane outage cannot come back up, though (see known limitations).
A per-cluster config repo. You register a cluster, hand it a token and values, and it installs itself. The plane’s registry is the list of clusters.

Topology

The control plane stores intent. The cluster stores truth. A cluster-resident operator reconciles the gap, and it always dials outbound.

                     ┌──────────────── control plane ────────────────┐
console ──http──►    │  api (python) ◄──► Postgres ◄──► hub (python) │
hgr     ──http──►    │                       ▲                 ▲     │
                     │                  reactor (python)       │     │
                     └─────────────────────────────────────────┼─────┘
                                                               │ WebSocket, dialled outbound
                                                               │
caller ──https──► gateway (go) ◄── HTTP/2 tunnel, dialled outbound ── agent (go) ──► model frontend
                     │                                               operator (go)
                     └── reads routing table from api           └─ one of each per cluster ─┘

Each cluster runs two Wingsmith processes:

Process	Job	Connection
operator	Receives desired state from the hub, renders it into Kubernetes objects, reports what it observes	WebSocket to the hub
agent	Carries inference traffic: proxies requests from the gateway to the model’s frontend in the cluster	HTTP/2 tunnel to the gateway

Neither the plane nor the gateway opens a connection into a cluster, and the plane never holds a kubeconfig. Multi-cluster systems such as Run:ai, Open Cluster Management, Karmada, Rancher Fleet and Argo CD’s agent mode converged on the same shape.

Why not push

The alternative is for the plane to hold credentials and call each cluster’s API server. It fails on five counts, and each gets worse as clusters are added.

Failure	Why it compounds
Credential blast radius. N clusters means N sets of cluster-admin credentials in one database	It is the most valuable target in the system
Reachability. Inbound to `:6443` is a firewall negotiation per cluster; outbound 443 works almost everywhere	Linear in clusters, forever
Partition behaviour. Under push, plane downtime means workloads stop being managed. Under pull, the cluster keeps scheduling, healing and serving	The difference between “the plane is down” and “inference is down”
Scheduling latency. A scheduler placing work over a WAN round trip to a remote API server is not a scheduler	GPU bin-packing needs local node state at millisecond scale
Rollout blast radius. A bad reconcile loop in the plane hits every cluster at once; an agent rollout is staged per cluster	Unbounded against bounded

Owning the clusters removes the reachability concern only. The other four are properties of the architecture, not of who owns the hardware.

Transport

The operator-initiated stream is a WebSocket carrying JSON envelopes: one connection, request/response plus server push, ordinary TLS, tractable backpressure and resume. It is not gRPC. The cost is that Go and Python types are written by hand, and a fixture corpus in libs/packages/contract/fixtures keeps them from drifting. A message type with no fixture is not part of the contract. See the wire protocol and ADR 0001.

Principles

Every object is declared, observed, or an operation

Kind	Written by	Lives
Declared	The plane; the cluster obeys	Plane database, synced down
Observed	The cluster; the plane reads	Synced up, denormalised into Postgres
Operation	Imperative, long-running, able to fail	Neither of the above

The third kind is the one that gets skipped. Drain is not a field on a node. It is cordon, evict, wait for in-flight requests, confirm, and possibly time out and force. It has a lifecycle, an actor, a reason and a failure mode. See operations.

The observed half is not a mirror of the declared half. Wingsmith never declares a Node, and observed objects get their identity from the side that observes them: the cluster mints the id when it first sees the object (ADR 0002).

Keeping the three apart lets the plane answer what was asked for, what is true, and what is being attempted as separate questions.

The wire carries domain objects, never Kubernetes manifests

The plane sends Wingsmith’s own Deployment, Pool and Operation. The operator expands them into whatever Kubernetes objects it currently favours. This is the most important structural decision in the system: the Kubernetes-level implementation can change without the plane learning anything, all Kubernetes knowledge stays on the cluster side, and a non-uniform fleet is the operator’s problem.

Scheduling splits into global and local

	Cluster selection	Device selection
Scope	Global, coarse, infrequent	Local, hot, frequent
Inputs	Capacity, device class, region, residency, cost	Bin-packing, topology, preemption
Lives in	The plane	The cluster

A plane that picked devices would need an accurate live model of every card in every region and would be wrong constantly. See quotas and placement.

Inference traffic never passes through the control plane

Inference traffic, logs, metric streams and model weights flow directly or through the edge, never through the control plane. If any did, control plane availability would become the inference SLA. A control plane is allowed to be down for an hour in a way an endpoint is not.

The gateway is the one Wingsmith process in the request path. It keeps serving from the last routing table it loaded when the plane is unreachable, but it needs the plane to start.

Processes

Control plane

libs/plane is one Python package that runs as three processes over one Postgres database.

Process	Job
api	HTTP request/response API, serving the console and `hgr`
hub	Terminates the long-lived operator streams, with resume and backpressure
reactor	Derives alerts on its own loop, under a lease, and reconciles derived configuration the fleet needs

They are separate because they fail differently: the api is stateless request/response, and the hub holds N persistent sessions.

Inside the package three concerns stay apart:

Modules are one package per object and the only place declared state is written.
Ingestion is the only writer of observed state: high-frequency, idempotent, last writer wins per cluster.
Placement writes nothing. It takes a snapshot of observed state and returns a decision, so it is testable without a cluster.

The hub scales by shedding session state instead of sharing it. Cursors and the replay buffer stay in one pod’s memory. Reconnecting to the same pod resumes exactly; reconnecting to another costs a full resync, which is expensive but always correct and never a silent gap. The one shared fact in Postgres is which hub session owns a cluster’s stream, claimed atomically at hello, and a displaced session notices by polling its own claim and closing. Correctness does not depend on sticky routing. See ADR 0005.

Operator

libs/operator runs once per cluster and has two loops:

Sync loop: stream to CRDs. Applies desired state from the plane into custom resources and ships observed state back up.
Reconcile loop: CRDs to cluster objects.

The CRDs are the durable local copy of desired state, which is what lets a cluster survive a plane outage. If the operator held desired state only in memory, a restart during a partition would make the cluster forget what it should be running. The resume cursor is persisted the same way, in a StreamCheckpoint resource.

Backpressure is fixed by the protocol. When the outbound queue is full, a newer frame replaces the oldest with the same type and subject, because observed state is a snapshot of the present and falling behind should cost resolution, not correctness. Events are dropped from the tail, with the loss counted. Liveness never depends on the hub session, so a partitioned operator is healthy.

Shipping

The control plane is a pet; inference clusters are cattle. The plane and the gateway are images you run yourself against a Postgres you operate and back up, and there is no chart for them yet. The operator and the agent are images each cluster pulls through the cluster-setup Helm chart on that cluster’s own helm upgrade. Registering a cluster mints its enrolment token and renders its values, and then you install the chart in it.

See control plane, enrol a cluster and cluster-setup chart.

Data plane

The gateway, tunnel and agent are how a request reaches a model. While running, none need the control plane.

The gateway takes an OpenAI-compatible request, reads model from the body and forwards it to a cluster serving that model. It keeps the routing table in memory and refreshes it on a timer.
The tunnel is HTTP/2 with the roles reversed: the agent dials out, then serves HTTP/2 on that connection while the gateway acts as client. Nothing dials into a GPU cluster.
Each model gets its own frontend in the cluster, and the gateway forwards the deployment id it resolved so the agent can pick the right one.

The exceptions are at startup: the gateway needs the plane to load its first routing table, and an agent that redials needs the gateway to verify its token, which the gateway answers from cache for up to 15 minutes if the plane is unreachable.

The full design, including failure behaviour, tenant identification and routing, is on data plane.

Known limitations

True today and worth knowing before you depend on it. Unbuilt features are on the roadmap.

See also security for the security posture of a deployment.