Operations and Events

Long-running operations (drain, scale, cordon), operation lifecycle, and event observation.

An operation is an imperative, long-running, failable action. Operations are neither declared state nor observed state. They’re the third kind of object: something you initiate that has a lifecycle, an actor, a reason, and a result.

Why operations exist

Some cluster management tasks are not fields. Drain is not a field on a node. It’s cordon, evict, wait for in-flight requests, confirm, and possibly time out and force. Modelled as node.status = 'draining', there’s no answer to “why is this stuck,” “who did this,” or “how long has it been.”

The same lesson arrives with GPU reset, node reboot, cache warm, rolling update, and workload scale. Each is long-running, can fail, and needs a record. One Operation object with a typed payload covers all of them.

Operation structure

Each operation has:

• type: What kind of operation (e.g., node.drain, workload.scale, device.reset)
• target: What it’s acting on (a node, workload, device, or cluster)
• actor: Who initiated it (the authenticated principal from the request)
• reason: Free text explaining why
• phase: Where it is in its lifecycle
• deadline: When it times out
• result: Outcome (when complete)
• payload: Type-specific parameters (e.g., {"desired": 8} for a scale)

Creating operations

You create operations via POST /v1/operations:

{
  "type": "node.drain",
  "target": "nod_01hq0v...",
  "reason": "firmware update",
  "deadline": "2026-10-28T12:00:00Z"
}

The target can be written two ways:

• A prefixed ID: nod_01hq0v..., dev_01hq0v..., wkl_01hq0v...
• A kind-qualified name: node:gpu-node-04, device:GPU-8f3c..., workload:llama-70b

If the name matches multiple objects, you get 422 invalid with the list of candidates. If it matches none, you get 404 not_found.

For targets inside a cluster, you can provide cluster in the body to narrow the match:

{
  "type": "node.drain",
  "target": "node:gpu-node-04",
  "cluster": "iad-1",
  "reason": "firmware update"
}

Lifecycle phases

An operation moves through phases:

1. pending: Created, not yet started.
2. dispatched: Sent to the cluster (for cluster-scoped operations).
3. running: In progress.
4. succeeded: Completed successfully.
5. failed: Completed with an error.
6. cancelled: Stopped before completion.
7. timed_out: Exceeded its deadline.

Plane-local operations (workload scale, workload pause, workload drain) skip dispatched and go straight to running, because they change declared state locally and don’t dispatch to a cluster.

Operation types

Node operations

node.drain: Cordon the node, evict pods, wait for in-flight requests to finish, confirm. Leaves the node cordoned but empty.

node.cordon: Mark the node unschedulable. No eviction. Existing pods stay.

node.uncordon: Mark the node schedulable again.

Device operations

device.reset: Reset one GPU. Requires draining workloads off it first.

Workload operations

workload.scale: Change the total replica count across all clusters.

Payload: {"desired": n, "min": n, "max": n} (all optional; omitted fields stay unchanged).

Scale is an operation, not a field write, because it’s long-running and can fail. A scale sits in pending phase for as long as quota withholds capacity. The operation’s phase and result answer “is this stuck on quota” and “has it landed.”

workload.pause: Set paused: true on the workload. Payload: {"paused": true | false}.

workload.drain: Remove the workload from one cluster. Payload: {"cluster": "clu_..."}.

workload.reshape: Flip the workload between aggregated (one worker role) and disaggregated (prefill + decode). Payload: {"topology": "aggregated" | "disaggregated", "desired": n, "roles": [...]}.

Reshaping reloads every worker with the new topology. The peak device count is old footprint plus new footprint while the old generation is still up. The result includes expected_unavailability so you can decide whether to take it now.

Deployment operations

deployment.roll: Force a rollout of the deployment, even if the generation hasn’t changed. Useful for pulling a new image or restarting workers.

deployment.benchmark: Run a load test against this deployment. Payload includes harness name and parameters. Creates a BenchmarkRun object and a deployment.benchmark operation. The run records the result; the operation carries it out.

Component operations

component.upgrade: Upgrade a component (operator, agent, Dynamo platform) in a cluster. Payload: {"component": "operator", "version": "v1.2.3"}.

Polling and results

Operations are long-running. You create them and poll GET /v1/operations/{id} for status. When complete, the operation’s result field holds the outcome:

• succeeded: The operation finished successfully. result.message may have details.
• failed: The operation failed. result.message explains why.
• cancelled: Stopped before completion via DELETE /v1/operations/{id}.
• timed_out: Exceeded its deadline.

Cancelling operations

DELETE /v1/operations/{id} cancels an operation that’s pending or running. The operation moves to phase cancelled. It doesn’t delete the record; you can still read it to see that it was cancelled and by whom.

Cancelling a completed operation does nothing and returns 204.

Plane-local vs cluster operations

Some operations change declared state in the control plane and re-run placement. They never dispatch to a cluster:

• workload.scale
• workload.pause
• workload.drain
• workload.reshape

These are plane-local. The cluster field is absent (for workload targets) or optional (for cluster targets). What reaches clusters is ordinary declared state with different shares or scale.

Other operations are dispatched to the cluster where their target lives:

• node.drain, node.cordon, node.uncordon
• device.reset
• deployment.roll, deployment.benchmark
• component.upgrade

The operator receives the operation, executes it, and reports the result back.

Events

An event is a point-in-time observation, not an action. Events are observed, read-only, and append-only. They’re logged by the operator or the plane when something noteworthy happens.

Examples:

• node.joined: A node appeared in the cluster.
• device.degraded: A device entered an unhealthy state.
• deployment.oom: A replica was OOMKilled.
• operation.timed_out: An operation exceeded its deadline.

Each event has:

• type: What happened
• subject: What it’s about (a node, device, deployment, etc.)
• cluster_id: Where it happened
• timestamp: When it happened
• detail: Structured payload with specifics

Events are read via GET /v1/events. You can filter by type, subject, cluster, and since.

Events feed alerting. The reactor evaluates alert rules against the event stream, deriving alerts when conditions are met.

Operations vs events

Use operations for actions you take. Use events for facts the system reports.