Clusters and Topology

Cluster enrollment, regions, capabilities, sync state, and multi-cluster coordination.

A cluster is one enrolled Kubernetes cluster with GPUs. Wingsmith places workloads across clusters, routes requests to them, and reads their observed state, but never holds credentials to reach into them. Each cluster runs two processes that dial outbound: hangar-operator (syncs desired state and reports observed state to the control plane) and hangar-agent (carries inference traffic to model frontends).

Enrollment

You register a cluster in the console or with hgr cluster create. Registration mints an enrolment token (prefix hgro1.), a per-cluster credential that authenticates both the operator and the agent. The operator presents it to the hub over WebSocket; the agent presents it to the gateway when establishing tunnels.

Once you install the cluster-setup Helm chart with the token in the cluster, the operator connects and begins syncing. From that point the control plane knows the cluster exists, tracks its reachability, and can place workloads on it.

Declared and observed state

Each cluster has two halves:

Declared: What you asked for. Fields include:

• phase: pending, active, draining, or disabled
• region: Geographic label for placement (e.g., us-east, eu-central)
• zone: Narrower location within a region
• residency: Data residency constraint (e.g., gdpr, us-only)
• metrics_label: The cluster label value this cluster’s metrics carry in the central metrics store

Observed: What is true right now. Fields include:

• reachable: Boolean, is the operator currently connected to the hub
• sync.state: connected, disconnected, syncing
• sync.last_sync_at: Timestamp of the last successful sync
• sync.lag_ms: How far behind the operator’s clock is from the hub’s
• operator_version: The hangar-operator version reported by the cluster
• capabilities: What the cluster supports (MIG, NVLink, RDMA)

These two halves never merge. A cluster can be declared active and observed unreachable at the same time, and both facts are true. The first says what you want; the second says the network is down. Folding them into one field would make those two situations indistinguishable.

Phases

The phase field controls whether placement uses the cluster:

• pending: Registered but not yet accepting workloads. New clusters start here.
• active: Normal operation. Placement considers it for new workloads, and existing deployments keep running.
• draining: No new workloads. Existing deployments stay, but the cluster stops taking new placements. Use this to empty a cluster before maintenance.
• disabled: Fully out of service. Placement ignores it, and its upstreams drop out of endpoint resolutions immediately. Agent tunnels are refused.

Changing phase to draining or disabled invalidates the gateway’s routing table, so new requests stop arriving within one poll interval (default 30 seconds).

Region and residency

region, zone, and residency are placement inputs. You can constrain a workload to specific regions or residencies via a Policy, and placement only considers clusters that match.

These are labels, not geographic facts. Wingsmith doesn’t validate that a cluster tagged eu-central is in Europe; you declare what each cluster is, and placement enforces your declarations.

Capabilities

The operator reports what the cluster supports in the capabilities object:

• mig: Does the cluster support NVIDIA MIG (Multi-Instance GPU)?
• nvlink: Are devices connected via NVLink/NVSwitch fabric?
• rdma: Is RDMA networking available?

Placement doesn’t act on these yet, but the operator reports them so future workload requirements can select clusters that have the needed fabric or partitioning.

Operator and agent

Each cluster runs two Wingsmith processes, installed by the cluster-setup chart:

hangar-operator: Connects to the hub via WebSocket. Receives desired state (deployments, pools, operations) and stores it in custom resources. Renders those CRDs into Kubernetes Deployments, Services, and Dynamo DynamoGraphDeployment objects. Reports observed state (nodes, devices, replicas, topology domains) back up to the control plane.

hangar-agent: Dials the gateway, authenticates with the cluster’s enrolment token, and opens HTTP/2 tunnels. When the gateway routes a request to this cluster, it forwards the request over the tunnel to the agent, which proxies it to the model’s frontend Service inside the cluster.

The two are separate processes with separate connections. If a request depended on the operator’s sync stream, control plane uptime would become the inference SLA. The agent keeps serving even if the operator is disconnected.

Sync state and partition behavior

The operator maintains a persistent WebSocket connection to the hub. If the network partitions:

• The operator keeps reconciling from its last-known desired state, stored in SyncedResource CRDs.
• The cluster keeps serving models. Existing replicas stay healthy.
• New deployments don’t arrive, because the plane can’t send them.
• Observed state stops flowing back, so the console shows stale replica counts.

When the network returns, the operator resumes from its saved cursor and syncs any updates it missed. The hub holds a replay buffer per operator session to cover short outages without a full resync.

A gateway that restarts during a control plane outage cannot load a routing table and stays down until the plane returns. Gateways that were already running keep serving from the last table they loaded.

One frontend per model

The operator renders one Dynamo frontend per deployment, not one shared frontend per cluster. This lets two deployments of the same model (a canary rollout, for instance) coexist without collapsing into one route.

The agent forwards requests by reading X-Hangar-Deployment-Id from the gateway and mapping it to the frontend Service labeled with that deployment ID. It doesn’t poll /v1/models, because that would silently merge deployments advertising the same model name.

Metrics label

Each cluster reports metrics to a central Prometheus or Grafana-compatible store via remote_write. The metrics_label field on the cluster registry entry is the value of the cluster label those series carry.

The control plane uses this label to query per-cluster series. If two clusters report the same label, their metrics are indistinguishable in the store. If a cluster’s label doesn’t match what it declares, the console shows no data for it.

Enrolling a cluster touches nothing outside the control plane: the operator reads its label from the hub, and the plane is the source of the remote_write credentials each cluster uses.