Concepts

Deployments and Replicas

Per-cluster placement, admission phases, authority, adoption, and replica observation.

A deployment is one workload on one cluster. It’s the unit the operator reconciles: the operator receives a deployment’s desired state from the hub, renders it into a Dynamo DynamoGraphDeployment CRD, and reports the observed replicas back.

Deployment basics

When placement expands a workload, it creates one deployment per cluster it chose. Each deployment carries:

workload_id: Parent workload
cluster_id: Which cluster this runs on
desired_replicas: This cluster’s share of the workload’s total scale
admission_phase: Where this deployment is in the admission flow
generation: Increments when the serving configuration changes
authority: managed or observed
enabled: Whether the operator should reconcile this

A deployment is plane-declared and cluster-observed. The plane writes what it wants (desired_replicas, enabled, serving config); the cluster reports what it did (status, replicas, live_digests).

Admission phases

Each deployment passes through admission before it runs:

pending: Just created, waiting for quota or capacity.
admitted: Quota check passed, cluster has room.
placed: Scheduler assigned devices, pods are being created.
running: At least one replica is ready.

A deployment moves from pending to admitted when:

The project’s quota for the needed device class allows it.
The cluster reports enough free devices of that class.

If either check fails, the deployment stays pending with a reason explaining why (e.g., insufficient_quota, insufficient_capacity).

Admission is per-cluster, not per-workload. A workload can be running in one cluster and pending in another. A workload’s overall phase is derived from its deployments: running if all are, failed if any failed, otherwise the least advanced phase.

Replicas

A replica is one observed running instance of a deployment on named devices. The operator reports replicas as they start, become ready, and terminate.

Each replica carries:

deployment_id: Parent deployment
name: Kubernetes pod name
node_id: Which node it runs on
device_ids[]: Device UUIDs allocated to this replica
ready: Is the replica serving traffic
serving_digest: Which serving configuration this replica is running
engine_version: Reported engine version

The ready count is what matters for routing. A deployment with desired_replicas: 4 and ready: 2 is degraded: it’s partially running but below target.

Generation and rollouts

A deployment’s generation increments when its serving configuration changes: a different engine version, context length, parallelism, or quantization.

When generation increments, the operator re-renders the Dynamo DynamoGraphDeployment. Dynamo rolls out the new generation while keeping the old one up, so the peak device count is old footprint plus new footprint. On a cluster sized for one, the new generation waits for the old to release devices.

During a rollout, live_digests shows both configurations and how many replicas are on each:

{
  "live_digests": {
    "svg_abc123": 2,  // Old generation, two replicas still up
    "svg_def456": 2   // New generation, two replicas ready
  }
}

Once all replicas converge on the new digest, the map has one entry.

Authority: managed vs observed

A deployment’s authority field controls whether the operator may write to the cluster:

managed: The operator reconciles. If desired_replicas is 4 and the cluster has 2, the operator creates 2 more.
observed: The operator only reads. It reports replicas, never writes. Recorded and checked for drift, but never acted on.

A deployment the plane created is managed from birth. A deployment adopted from a running workload starts as observed and can be promoted to managed later.

Why authority exists

Wingsmith is pointed at clusters already running models. Importing them as declared state would restart every one the first time the reconciler ran. So the operator reports what it finds as an observation, and tracking it is a decision to observe, not to manage.

Promoting a deployment to managed is a separate, explicit act: POST /v1/deployments/{ref}/authority with {"authority": "managed"}. Before you promote, you review the imported configuration against the live cluster to confirm they match.

Adoption workflow

When the operator finds a workload running in a cluster that the plane never declared, it reports it as discovered. The plane records the sighting under /v1/discovered and shows it to you.

You decide what to do:

Track it: POST /v1/discovered/{ref}/track. Keeps it in the inventory as observed only. No deployment is created yet.
Ignore it: POST /v1/discovered/{ref}/ignore. Marks it as not interesting. Drops out of the default list.
Promote it: POST /v1/discovered/{ref}/promote. Creates a workload pinned to this cluster, with one deployment set to authority: observed. Now it’s a real workload you can edit, scale, or promote to managed later.

A promoted deployment stays observed until you explicitly promote its authority. That’s two separate decisions: “catalog this” and “take control of it.”

Imported arguments

When a deployment is adopted, the operator reads the engine’s argv from the pod spec and records it as imported_args. This is the full flag list the engine is running with.

imported_args stays separate from engine_args. Reading a flag off a running pod records it; asserting it as desired state is a separate act. When you promote the deployment to managed, you review imported_args and decide which flags to keep in engine_args.

Deployment status

Each deployment has a status object, which is the cluster’s answer:

phase: What the cluster reports (distinct from admission_phase, which is the plane’s decision)
reason: Why it’s in that phase (e.g., PendingWeightDownload, InsufficientDevices)
message: Human-readable explanation
ready_replicas: Count of ready replicas
desired_replicas: What the cluster is trying to achieve (echoes the deployment’s desired_replicas)

status is null until the cluster reports at all. That’s every deployment’s first few seconds and the permanent state of a deployment whose cluster can’t reach the hub.

admission_phase is the plane’s decision about this cluster. status.phase is what the cluster actually did. They’re separate because they have separate writers.

Replica allocation and devices

When a replica starts, the Kubernetes scheduler assigns it to a node and claims device UUIDs. The operator reports this as an allocation: which replica holds which devices.

Each allocation has:

replica_id
device_ids[]
node_id

The control plane reads allocations to:

Count how many devices a project is using (for quota enforcement).
Answer “which devices does this replica use.”
Check that the replica got devices in one topology domain if the workload required it.

A device is allocated when it appears in a live allocation. An allocated device doesn’t count as free capacity for placement.

Rendering to Dynamo

The operator renders each deployment into one NVIDIA Dynamo DynamoGraphDeployment CRD. That CRD describes:

One frontend per deployment (labeled with the deployment ID, so the agent can route to it)
Worker roles (prefill, decode, or one aggregated role)
Parallelism (TP/PP/EP)
Device requests
KV cache settings

Dynamo handles the rest: scheduling workers onto GPUs, starting the engine processes, routing between frontend and workers, scaling roles independently (for disaggregated serving).

Wingsmith never sends Kubernetes manifests. It sends its own Deployment object. The operator translates that into whatever Kubernetes and Dynamo objects it currently favors. This is what lets the fleet be non-uniform and lets renderer implementations change without the plane learning anything.

Read-only mode

If the operator is in read-only mode (HANGAR_OPERATOR_READ_ONLY=true), it reconciles nothing. It still syncs down desired state and stores it in CRDs, and it still reports observed state back up. But it never writes to the cluster: no DGDs created, no pods modified.

Use this to test a plane-operator connection without touching a production cluster.