Drain and Scale Operations

Drain nodes and workloads, scale workloads, cordon nodes; operation lifecycle and waiting.

Operations are imperative, long-running, failable actions that change the fleet or workloads. They have a lifecycle with phases, are recorded with who asked and why, and can be waited on or cancelled.

Operation types

Node drain

Drain a node to remove all workloads from it, typically for maintenance.

hgr operation create node.drain <node-name> --reason "firmware upgrade"

Or in the console, from Fleet → Nodes, select the node and click Drain.

Phases:

1. pending: operation created, not yet started
2. running: node is cordoned, replicas are being evicted
3. succeeded: all replicas evicted and rescheduled
4. failed: could not complete (e.g. no capacity to reschedule)

The node remains cordoned after the drain succeeds. Uncordon it manually when maintenance is done:

hgr operation create node.uncordon <node-name>

Wait for completion

--wait blocks until the operation finishes:

hgr operation create node.drain iad-1-n02 --reason "firmware upgrade" --wait

The CLI exits 0 if the operation succeeds, or exit 11 if it fails or is cancelled.

Drain result

A successful drain reports evicted_replicas=N in the result. If some replicas could not be rescheduled, the operation fails with details in the result message.

Node cordon

Cordon marks a node unschedulable without evicting existing replicas. Use this as a first step before drain, or to prevent new work on a node you suspect is unhealthy.

hgr operation create node.cordon <node-name>

In the console, from the node’s detail page, click Cordon.

Cordoned nodes show a badge on the node list. New replicas are not placed there, but existing ones keep running.

Workload scale

Change the number of replicas for a workload globally. Wingsmith distributes the new scale across clusters according to placement policy.

hgr workload scale <workload-name> --replicas 4

Or in the console, from Serving → Workloads, select the workload and click Scale.

A scale operation:

• Is recorded with a reason (optional but recommended)
• Can be waited on with --wait
• Updates every deployment of this workload

After the scale succeeds, each cluster’s operator reconciles the new desired state into its DynamoGraphDeployment. Replicas are added or removed as needed.

Immediate vs operation

Workload scale in the console or via hgr workload scale creates a workload.scale operation. Direct mutation via PATCH /v1/workloads/{id} is immediate and creates no operation record. Use operations for audited changes; use direct mutation for API-driven automation that tracks changes elsewhere.

Workload pause and resume

Pause scales a workload to 0 replicas without deleting it. Resume restores the scale before pause.

hgr workload pause <workload-name> --reason "nightly shutdown"

The workload’s state becomes paused. The scale before pause is remembered.

Resume:

hgr workload resume <workload-name>

The workload returns to its previous scale. Pausing and resuming are both recorded as operations.

Use cases:

• Nightly or weekend shutdown to save costs
• Maintenance windows
• Controlled rollout (pause old version, deploy new, resume new)

Workload drain

Drain evicts all replicas of a workload from one cluster without changing the global scale. Use this to migrate a workload off a cluster.

hgr operation create workload.drain <workload-name> --cluster <cluster-name> --reason "cluster decommission"

In the console, from the workload’s detail page, select the deployment for the cluster and click Drain.

After the drain succeeds, the deployment on that cluster has 0 replicas. The workload’s global scale is unchanged; other clusters still serve it. To remove the deployment entirely, delete the workload from the cluster via the console or API.

Operation lifecycle

Every operation progresses through phases:

1. pending: created, queued, not yet started
2. running: in progress
3. succeeded: completed successfully
4. failed: could not complete (see result message for why)
5. cancelled: stopped by user before completion

Operations are state machines. Only pending and running operations can transition further; succeeded, failed, and cancelled are terminal.

View operation history:

hgr operation ls

Filter by type, target, or outcome:

hgr operation ls --type node.drain --phase succeeded
hgr operation ls --target <node-name>

Cancel an operation

Cancel a pending or running operation:

hgr operation cancel <operation-id>

Or in the console, from Activity → Operations, select the operation and click Cancel.

A cancelled operation moves to cancelled phase. Any changes already applied (e.g. a node already cordoned) are not rolled back. The operator stops further actions for that operation.

Wait on an operation

--wait blocks the CLI until the operation finishes:

hgr operation create node.drain iad-1-n02 --reason "firmware" --wait

Exit codes:

• 0: operation succeeded
• 11: operation failed or cancelled
• 8: wait timeout (use --wait-timeout to extend)

In code, poll GET /v1/operations/{id} until phase is terminal, or use a WebSocket listener if available.

Operation result

A terminal operation has a result object with:

• message: human-readable outcome
• attempts: how many times the operation was retried
• duration: wall time from pending to terminal phase

For example, a successful node drain:

{
  "id": "op_01...",
  "type": "node.drain",
  "target": "iad-1-n02",
  "phase": "succeeded",
  "result": {
    "message": "evicted_replicas=3",
    "attempts": 1,
    "duration": "42s"
  },
  "reason": "firmware upgrade",
  "created_by": "admin",
  "created_at": "2026-10-27T12:34:56Z"
}

Idempotency

Operations use an idempotency key to prevent duplicate execution. The CLI generates a fresh key per invocation by default. To retry the same operation:

hgr operation create node.drain iad-1-n02 \
  --reason "firmware" \
  --idempotency-key custom-key-123

Sending the same key within its validity window (typically 24 hours) returns the existing operation instead of creating a duplicate.

Limitations

• No gang operations yet. Draining multiple nodes requires one operation per node. Gang operations (one operation, many nodes) are on the roadmap.
• No rollback. A cancelled or failed operation does not undo changes already applied. For example, a failed drain leaves the node cordoned; uncordon it manually.
• No progress tracking. Running operations do not report progress (e.g. “3 of 5 replicas evicted”). They are either running or done.

Example: drain a node for maintenance

Drain the node:

hgr operation create node.drain iad-1-n03 --reason "replace faulty GPU" --wait

Wait for success (the CLI blocks with --wait, or poll until phase: succeeded).

Perform maintenance (replace hardware, reboot, etc.).

Uncordon the node:

hgr operation create node.uncordon iad-1-n03

Verify the node is schedulable:

hgr node get iad-1-n03

Check that cordoned: false.

Example: scale a workload down and back up

Scale down:

hgr workload scale llama-3-8b --replicas 1 --wait

Deployments reconcile to 1 replica each. Verify:

hgr deployment ls --workload llama-3-8b

Scale back up:

hgr workload scale llama-3-8b --replicas 4

Troubleshooting

Drain stuck in running

A drain may hang if:

• No capacity to reschedule evicted replicas (quota exhausted, or no devices available)
• Replicas are pinned by a PodDisruptionBudget (not yet supported by Wingsmith)

Check device availability:

hgr device ls --cluster <cluster> --allocatable

Check quota:

hgr quota ls --project <project>

Cancel the operation and address capacity before retrying:

hgr operation cancel <operation-id>

Cordon fails with forbidden

The operator’s RBAC may not grant patch on nodes. Check the chart’s cluster.allowedOperations:

cluster:
  allowedOperations:
    - cordon
    - drain

On a read-only cluster, operations are refused.

Operation shows failed with no message

Check the operator logs:

kubectl logs -n hangar-system deployment/hangar-operator

The operator logs the reason for failure. The operation’s result.message should also contain details.

Next steps

• Operations and Events for the object model
• Operator configuration for RBAC and allowed operations
• Roadmap for planned operation features (gang operations, autoscaling)