Wingsmith
wingsmith.dev
Serving

Adopt Existing Deployments

Track discovered workloads, review imported config, promote to managed, set authority.

Adoption lets you track models already running on a cluster without immediately taking control of them. The operator discovers workloads, Wingsmith records them as observed, and you decide whether to manage them.

How discovery works

The operator reports workload.discovered events for any DynamoGraphDeployment in the workload namespace that Wingsmith does not recognize. A discovered workload is a model running outside Wingsmith’s control.

Discovery runs on every sync. When a new graph appears, the operator sends:

  • The graph’s name
  • Replica count
  • Engine arguments (as imported_args, read-only)
  • Observed device allocation

The plane creates a discovered workload record with authority: observed.

Track a discovered workload

In the console, go to Serving → Discovered. The table lists workloads the operator found but Wingsmith does not own.

For each discovered workload:

  • Name: derived from the Dynamo graph name
  • Cluster: where it was discovered
  • Replicas: observed count
  • Imported args: engine arguments from the existing graph
  • Authority: always observed at discovery

To track it, click Track. Wingsmith creates an observed deployment for this workload. The deployment is recorded in the plane’s database and appears under Serving → Deployments with authority: observed.

With hgr:

hgr discovered ls
hgr discovered track <discovered-id>

Tracking does not change anything on the cluster. The operator continues to observe the graph; it does not reconcile it.

Authority: observed vs managed

Authority determines whether Wingsmith observes or controls a deployment.

  • observed: Wingsmith records what the operator reports but never changes the cluster. The deployment is read-only. Changes to the graph (scale, args, engine version) made outside Wingsmith are reflected in the plane.
  • managed: Wingsmith owns the deployment. The operator reconciles declared state into the cluster. The plane may scale, update, or delete the graph.

A discovered workload starts as observed. Promotion to managed is explicit.

Review imported configuration

Before promoting a discovered workload, review its imported configuration. The plane extracts:

  • Engine: vLLM, SGLang, or TensorRT-LLM
  • Engine version: from the image tag
  • TP / PP / EP: parallelism degrees inferred from worker count and device topology
  • Imported args: the existing engine_args or frontend_args, stored as a read-only snapshot

The plane does not attempt to create an artifact for the weights. You must publish an artifact manually and associate it with the workload if you promote it.

In the console, from the discovered workload’s detail page, click Review Config. Compare the imported args against Wingsmith’s declared format. Args that conflict with Wingsmith’s rendering (e.g. --served-model-name, --disable-log-requests) are highlighted.

Promote to managed

Promotion converts an observed deployment into a managed one. This is a one-way decision: once managed, the deployment’s authority cannot be changed back to observed via the console (though the API allows it).

From the discovered workload’s page, click Promote to Managed. The console prompts:

  1. Create or link artifact: you must provide an artifact reference. If the weights already exist in Wingsmith’s catalog, select the artifact. Otherwise, publish the artifact first.
  2. Review config diff: the plane shows what will change when it takes over. Imported args are merged with Wingsmith’s defaults; conflicts are flagged.
  3. Confirm promotion: the workload’s authority changes to managed. The operator renders a new DynamoGraphDeployment from Wingsmith’s declared state, replacing the existing graph.

With hgr:

hgr discovered promote <discovered-id> --artifact <artifact-id>

After promotion:

  • The deployment appears under Serving → Deployments with authority: managed.
  • The operator reconciles Wingsmith’s declared state. The existing graph is deleted and re-rendered.
  • The model reloads its weights. Serving is interrupted briefly during the transition.

Imported args vs declared args

After promotion, the deployment has two sets of arguments:

  • Imported args: the original engine_args from discovery, stored read-only as a reference
  • Declared args: engine_args declared in Wingsmith, which the operator renders

Imported args are never applied by the operator. They are preserved for comparison only. Declared args are what the cluster runs.

If you want to preserve specific args from the imported config, copy them into the workload’s declared engine_args before or after promotion. The console flags args that differ between the two sets.

Authority and scaling

An observed deployment cannot be scaled via Wingsmith. Scale changes made outside Wingsmith (e.g. by editing the graph directly with kubectl) are reflected in the plane’s observed state.

A managed deployment can be scaled via hgr workload scale or the console. The operator reconciles the new scale into the cluster.

Deleting a discovered workload

Before promotion, you can ignore a discovered workload:

hgr discovered ignore <discovered-id>

The plane removes the record. The operator continues to report it on every sync, so it reappears unless you also delete the graph from the cluster.

After promotion (now managed), delete the workload via:

hgr workload rm <workload-id>

The operator deletes the DynamoGraphDeployment and evicts all replicas.

Restoring an ignored workload

If you ignored a discovered workload and want to track it again:

hgr discovered restore <discovered-id>

Or re-track it from the Discovered table in the console. The operator reports it again on the next sync.

Example: adopt a running model

A model gpt-j-6b is running on cluster iad-1, deployed outside Wingsmith.

  1. The operator discovers it and reports workload.discovered.

  2. In the console, go to Serving → Discovered. The model appears in the table.

  3. Click Track. Wingsmith creates an observed deployment for gpt-j-6b on iad-1.

  4. Click Review Config. The imported args show --model gpt-j-6b --tensor-parallel-size 2.

  5. Publish an artifact for GPT-J-6B weights (if not already in the catalog):

    hgr artifact create gpt-j-6b \
  --digest sha256:... \
  --location hf://EleutherAI/gpt-j-6b \
  --format safetensors

  6. Click Promote to Managed, select the artifact, review the diff.

  7. Confirm. Wingsmith deletes the old graph and renders a new one from declared state. The model reloads.

  8. The deployment now appears under Deployments with authority: managed. Scale or update it via Wingsmith.

Limitations

  • No artifact inference. The plane does not extract the model’s weights location from the existing graph. You must provide the artifact manually.
  • No zero-downtime transition. Promotion deletes the old graph and creates a new one. The model reloads and serving is interrupted.
  • Imported args are read-only. After promotion, imported args are not applied. You must copy desired args to the declared config.
  • No bulk adoption. Each discovered workload must be tracked and promoted individually. Bulk operations are not supported yet.

Security note

Tracking a discovered workload does not grant Wingsmith permission to change it. Until promoted to managed, the deployment remains under the control of whoever deployed it.

After promotion, Wingsmith owns the graph. The operator renders it and can delete it. Treat promotion as a transfer of ownership.

Troubleshooting

Discovered workload not appearing

The operator only discovers graphs in the workload namespace configured in the chart (workloadNamespace.name, default dynamo). Graphs in other namespaces are ignored.

Check the operator logs:

kubectl logs -n hangar-system deployment/hangar-operator

Promotion fails with artifact_not_found

The artifact ID provided does not exist. Publish the artifact first:

hgr artifact create <name> --digest <digest> --location <url> --format <format>

Imported args conflict with declared args

The console flags conflicting args during promotion. Common conflicts:

  • --served-model-name: Wingsmith sets this to the endpoint name. Remove it from imported args.
  • --disable-log-requests: Wingsmith enforces retention policy. Remove logging args.

Copy safe args (e.g. --max-num-seqs, --gpu-memory-utilization) into the workload’s engine_args if you want to preserve them.

Promotion interrupted during graph render

If the operator is restarted during promotion, the graph may be partially rendered. Check the graph status:

kubectl get dynamographdeployment -n dynamo

If the graph is stuck, delete it manually and retry the promotion:

kubectl delete dynamographdeployment <graph-name> -n dynamo
hgr discovered promote <discovered-id> --artifact <artifact-id>

Next steps