Plane-Operator Wire Protocol

WebSocket + JSON envelope, connection lifecycle, resume, backpressure, versioning.

One WebSocket, dialled outbound by the operator, carrying desired state down and observed state up. Because the transport is hand-written (WebSocket + JSON instead of gRPC), four things are Wingsmith’s code instead of a library’s: framing, correlation, resume, and backpressure.

Connection lifecycle

operator                                            hub
   │                                                 │
   ├─ WSS connect (Authorization: Bearer <token>) ───►
   │                                                 │
   ├─ hello { cluster_id, operator_version, ... } ──►
   │                                                 │
   ◄─ welcome { session_id, ack_seq, full_resync } ─┤
   │                                                 │
   ├─ ... frames both directions, indefinitely ... ─►
   ◄──────────────────────────────────────────────────┤
   │                                                 │
   ├─ ping (every 10s) ──────────────────────────────►
   ◄─ pong ───────────────────────────────────────────┤

hello is always the first frame and is never sent again on the same connection. The hub answers with welcome or closes with a typed error. A connection that has not completed the handshake within 10 seconds is closed.

Authentication

Each cluster authenticates with its own enrollment token. The bearer token on the connect is the cluster’s enrollment token, issued by the plane when the cluster is registered and stored there only as a hash. There is no shared or fleet-wide operator credential: anything not shaped like an enrollment token is turned away before the handshake.

The token, not hello, decides which cluster the session speaks for: hello.cluster_id must name that same cluster, by uid or by name, or the hub closes with a protocol error.

Whether the cluster may hold a session is asked at hello and again on every ownership poll for as long as the session lives. A cluster that is not registered, has been disabled, or whose enrollment token was rotated after this session opened is refused with an error and a 4403 close, so disabling a cluster or rotating a leaked token cuts a live session instead of waiting for the operator to reconnect.

Version checking

hello.operator_version is checked against the plane’s release line. A release-shaped version (0.4.2, v0.4.2) outside the major the plane speaks is refused with retryable: false and a protocol-error close, because a different major is free to change what a frame means. A version that is not release-shaped (dev, a commit) is a build from source and is admitted, with a log line on the hub.

Envelope

Every frame is a single JSON object. There are no bare payloads.

{
  "v": 1,
  "type": "device.observed",
  "id": "01J9X2E5G7QK8V3B0YHN4TZC6M",
  "seq": 4412,
  "ack": 4390,
  "reply_to": null,
  "ts": "2026-08-19T14:02:11.482Z",
  "payload": {}
}

Unknown fields are ignored on receipt. Unknown type values are never fatal: this is what lets an operator run one version behind the plane. The frame is dropped, and the receiver answers with an error naming the type and retryable: false. Dropping in silence would make version skew invisible. The error is an ordinary sequenced frame; answering an unknown type therefore consumes a sequence number.

A dropped frame is still counted. A receiver that cannot decode a payload (unknown type, malformed body) must still advance its arrival expectation for that seq and acknowledge it. Otherwise the next frame arrives as a gap and closes the connection. Only the envelope is fatal when undecodable; the payload is not.

An error frame is never fatal to an established session, whatever its retryable value. Fatality is reserved for the handshake, where a rejected credential leaves nothing to continue.

Correlation

Three interaction shapes, all on one connection:

• Event: fire and forget. reply_to is null and no response is expected. Most observed state is this.
• Request/response: the responder sets reply_to to the request’s id. Requests carry a deadline in their payload; a responder that cannot meet it must answer with error, not go quiet.
• Stream: a sequence of frames sharing one reply_to, terminated by a frame of type stream.end. Used for full resync and log-shaped data.

There is exactly one generic failure frame:

{ "type": "error", "reply_to": "<id>", "payload": {
    "code": "not_found | invalid | conflict | unavailable | internal | deadline",
    "message": "human readable, never parsed",
    "retryable": true }}

Sequence numbers and resume

Each direction numbers its own stream from 1 and continues across reconnects within a resumed session. Both sides hold a replay buffer of sent frames.

Not every frame is sequenced. hello, welcome, ping, pong, and ack carry no seq and never enter a replay buffer. Everything else is sequenced, including stream.end: a prune decision depends on that frame, so it must not be able to arrive out of order or go missing quietly.

ack exists for the case where a side must release its peer’s replay buffer but has nothing else to send. Under one-way load the heartbeat would otherwise be the only carrier, and holding a buffer for up to a full ping interval pins needless memory.

Acknowledgement is continuous, not per-session

Every frame may carry ack: the highest contiguous seq the sender has durably processed. Receiving it releases everything at or below it from the peer’s replay buffer. Contiguous is the operative word: a hole is never acknowledged past, so a frame whose handler failed is simply not acked and will be replayed. Withholding the ack is how a receiver signals “I could not apply this”: there is no negative acknowledgement.

Reading does not stop at a hole. Frames after a failed one are still delivered and applied (they are idempotent and delivery is at-least-once), but the contiguous cursor stays behind the hole, so nothing past it is acknowledged. This makes the receiver’s arrival expectation and its applied cursor two different numbers.

On resume, a receiver must rewind its arrival expectation to ack_cursor + 1. The peer is about to replay from the last acknowledged point, and a receiver still expecting the higher number it had already seen would read that replay as a duplicate, drop it, and never advance its cursor again. The session survives and makes no progress, which is worse than a clean failure.

Full resync

On reconnect the operator sends hello with resume_from set to its highest contiguous processed seq. The hub answers welcome with its own ack_seq, and either resumes the session or returns a new session_id with full_resync: true when the gap exceeds its buffer.

Three conditions force full_resync, and resume_from is not consulted for any of them: the session_id is one the hub has no record of (a hub restart looks exactly like this), lossy is set, or the requested point has already left the replay buffer. Unreleased frames in the sender’s replay buffer are re-sent immediately after welcome.

When the hub grants a fresh session (full_resync: true, or a session_id different from the one asked for) both directions restart numbering at 1 and both replay buffers are discarded. Frames held under the old numbering are not re-sent; the resync supersedes them.

full_resync means: the hub streams its complete desired state for that cluster, the operator streams its complete observed state back, and both sides then discard anything absent from that snapshot. Each side opens its own stream explicitly with resync.begin and closes it with stream.end: the full_resync flag announces this is about to happen but does not stand in for the framing. Without the explicit open, a receiver could not tell a resync snapshot from ordinary traffic, and the prune at the end is only safe against a stream it knows the boundaries of.

Resync is correct but expensive. The buffer is sized to cover a rolling restart, not a network hiccup. lossy: true in hello means “I cannot serve as a resume base”, and the sender must be given a fresh session instead of being allowed to pretend it can resume. session_id in hello names which session is being resumed: resume_from alone cannot, since a sequence number means nothing without the session it counts within.

Sequenced frames are processed strictly in order. A duplicate (seq below the expected) is dropped. A gap (seq above the expected) closes the connection instead of being accepted as a hole.

The resume cursor must be durable. It lives beside the CRDs, not in memory. An operator that restarts during a partition and comes back with an in-memory cursor rewinds its own stream, which forfeits the property the whole architecture exists to provide.

Resume is per hub instance, not per fleet

The hub runs N replicas and the session (cursors and replay buffer) lives in one replica’s memory, never in Postgres. Reconnecting to the same pod resumes exactly as described above. Reconnecting to a different pod (which a Kubernetes Service makes ordinary) finds a pod with no record of the session and therefore full_resync: true; this is expensive but never wrong.

What is durable across every pod is ownership: one hub session is the writer for a given cluster_id, established by an atomic claim in Postgres at hello time, so two sessions can never both believe they hold the same cluster. A hub instance that loses its claim to a newer hello elsewhere discovers this on its own (nothing pushes it a notice) and closes the connection.

Because that discovery is by polling, displacement is bounded, not instantaneous: for up to one poll interval the displaced session may still write. It is by construction the older and therefore staler of the two, so what that window can produce is a stale row and a flap of reachable, never a divergence that survives.

Liveness

Application-level, not WebSocket ping frames: those are handled by intermediaries and prove nothing about the peer’s event loop.

The operator sends ping every 10s. A peer that has seen no frame of any kind for 30s considers the connection dead and closes it. Reconnect backoff is exponential from 1s to 30s with full jitter. There is no maximum retry count; an operator that cannot reach the hub keeps reconciling from its CRDs indefinitely.

Backpressure

Both sides hold a bounded send queue. The policy differs by direction and the asymmetry is deliberate:

• Operator → hub: Observed state is a snapshot of the present, so when the queue is full the oldest frame of the same type and subject is replaced, not appended. Falling behind degrades resolution, never correctness. Events (*.event) are exempt: they are dropped from the tail and the drop is counted and reported, because a silently lost event is worse than a known-lost one.
• Hub → operator: Desired state must not be dropped. When the queue is full the hub closes the connection and lets resume sort it out. Correctness over continuity.

When the queue is full and no frame matches the incoming one’s type and subject, the oldest frame overall is evicted and counted as a drop. This is recoverable for the same reason the replacement case is: the sender will observe the state again.

Sequence numbers are assigned at write time, not when a frame is queued. A queue that renumbers nothing can replace and evict freely; a queue holding pre-numbered frames cannot, because every replacement leaves a hole that the receiver is required to read as a gap and close the connection over.

Poison frames

Withholding the ack asks for a replay, so it’s only the right answer when a replay could succeed: a database that was down, a lock that was held, a dependency not yet written. A frame that can never be applied is different: a malformed id, a payload that violates a stated invariant. There is no negative acknowledgement to express that with, so withholding the ack on one would replay it forever and the stream behind it would never advance.

A permanently unappliable frame is therefore accepted, discarded, and logged loudly, exactly as an unknown type is. The receiver counts the seq, acks it, and records the rejection with enough detail to identify the sender. This is the one place the protocol trades a lost record for liveness, and it’s deliberate: one dropped observation is a stale row until the next resync, while a wedged stream is a cluster that has stopped being managed.

Transient failure withholds. Permanent failure accepts and drops. A receiver that cannot tell the two apart should withhold, because a replay loop is at least visible.

What must never cross this link

Inference traffic. Logs. Metrics. Model weights. If any appear in a payload, the control plane’s availability has become the inference SLA, and the architecture has been lost. Telemetry goes from cluster to observability stack directly; this link carries state, not measurement.

Message types

Payload schemas live in MESSAGES.md, one canonical example each in fixtures/. A type with no fixture is not part of the contract, and both sides run a test that round-trips every fixture. That corpus is the only thing standing in for codegen.

Session

Supply: operator → hub

Device metrics (utilization, temperature, throughput, memory in use) are not here. Device memory appears only as memory_total_mb: capacity is state, occupancy is measurement.

Demand: hub → operator

The scale bounds on deployment.desired are this cluster’s share, never a workload total. A workload declares totals across every cluster it may use; the plane divides them and each operator receives only its own portion. An operator sees one cluster and cannot tell a whole workload from a fragment of one. A receiver renders exactly the number it’s sent and must never sum, scale, or re-derive it against anything else.

The serving configuration travels inlined, on deployment.desired.serving. It used to be a declared object of its own, recipe.desired, referenced by recipe_id. The separate frame bought an ordering hazard. A deployment could arrive naming a configuration that had not landed and wait for it, and after a resume that replayed the deployment but not the recipe it waited forever. Inlining removes the frame, the ordering rule, and the wait.

Demand: operator → hub

deployment.observed carries replica_ids, and it’s how the plane forgets a replica while the deployment still exists. A replica that simply stops being reported (a rolled pod, a restarted one, a deployment scaled down) has nothing else to remove its row. The frame names this deployment’s whole replica set on this cluster every time it’s sent, and the plane deletes the ones it doesn’t name.

Operations

Operations are the third kind of thing: imperative, long-running, able to fail. They are not status fields on the objects they act upon.

Operations are outside the resync contract. A full-resync snapshot is the authority for pruning declared state, and an operation absent from one has not been withdrawn: it’s not the kind of thing a snapshot describes. Pruning against one would cancel work already half-done, so an operation is accepted once, run to a conclusion, and reported. Delivery is at-least-once with no replay window: a receiver drops a redelivery of an operation it’s already running.

Resync

reply_to is the sole correlation between a resync.begin and its terminating stream.end. A receiver handling concurrent resyncs of different scopes matches them on that and nothing else.

A resync opens with resync.begin and closes with the ordinary stream.end; there is no resync.end. Everything between is ordinary .desired / .observed frames sharing one reply_to. There is no special resync encoding: a snapshot is just the whole set, sent at once. stream.end carries complete, and a snapshot that does not end with complete: true must not be pruned against.

Changing this protocol

Additive payload fields are safe and need no version bump. Removing or retyping a field, or changing envelope semantics, requires a new v and a period where both versions are spoken. Since the hub is upgraded before the operators are, the hub is always the side that carries compatibility code.

Each side writes one envelope version and reads a range. The plane and the operator each hold the version they write and the oldest they still read (ENVELOPE_VERSION and MIN_ENVELOPE_VERSION in libs/plane/app/hub/messages.py, EnvelopeVersion and MinEnvelopeVersion in libs/operator/internal/wire/envelope.go), and accept any frame whose v falls in [min, current]. Today both are 1. A frame outside the range closes the connection.