Architecture decisions

Every architecture decision record for Wingsmith: what was decided and why.

Architecture decision records (ADRs) capture the choices that shaped Wingsmith, why they were made, and what they cost. This page covers all eighteen. The full records, with context and alternatives, are in docs/decisions/ in the repository.

ADR 0001: WebSocket and JSON over gRPC

Decided: The plane-operator stream is WebSocket + JSON, kept in step by fixtures, not gRPC with generated types.

Why: At tens of clusters, gRPC’s value was never throughput. It was framing, request/response correlation, flow control, and mechanical protection against Go and Python types drifting. Its cost was a protoc/buf toolchain in every build, an unreadable wire during the phase where the architecture is being falsified, debugging that means learning gRPC’s keepalive semantics instead of reading Wingsmith’s own code, and generated Python that sits badly next to msgspec.

The message count is moderate and the codebase is fluent in msgspec and JSON. Being able to tail a connection and read what’s on it is worth more right now than efficiency Wingsmith doesn’t need. So Wingsmith takes the trade knowingly and writes framing, resume, liveness, and backpressure itself.

Consequences:

• libs/packages/contract/ is the single source of truth and no component may edit it unilaterally.
• Every message type ships with a fixture in the same change.
• Drift fails in CI, not silently in production. A message type with no fixture is not part of the contract.
• The escape hatch: if the message set passes roughly fifty types, or throughput ever becomes real, the payload encoding can be swapped to protobuf over the same WebSocket. The envelope, connection lifecycle, and resume semantics do not change.

ADR 0002: Clusters mint observed ids

Decided: The cluster mints the ids of the objects it observes; the plane stores the id as given and never re-mints it.

Why: Observed objects need a handle from the moment they’re discovered, and the plane cannot supply one for a thing it has not yet heard of. The alternative, the plane assigning ids on first ingestion, means the operator reports a thing by a local identifier, the plane mints a global one, and the two are joined per cluster forever. Every subsequent reference then costs a lookup, or the operator stores the plane’s id back to disk, which is observed state flowing down instead of up.

Consequences:

• node.observed carries the nod_ id, not a name the plane will later assign an id to.
• device.observed names its node by that same id.
• Id shape is contractual. An observed id is its prefix followed by a Crockford base32 ULID body, and a frame carrying a malformed one is not something a replay can fix.
• uid is written on insert and appears in no update set, so a row rediscovered after a reconnect keeps the id it was first given.
• A frame carrying a malformed id is a poison frame: it is accepted, discarded and logged, because withholding the ack would replay it forever.

ADR 0003: Telemetry is read beside the API, not through the hub

Decided: The plane reads telemetry from the metrics stores directly, at request time. None of it passes through the hub, and the browser never queries a store.

Why: Utilisation, temperature, throughput and latency are sampled at a rate about three orders of magnitude off the rate at which inventory changes. Pushing them through the sync channel would make the channel about metrics. Keeping measurement on its own surface also means a metrics store that is down cannot make a node page fail, and absent is never zero: a missing series drawn as 0% reads as an idle GPU, which is the one wrong answer that looks right.

Consequences:

• Metric names are Wingsmith’s own and vendor-neutral. Dialects are a table.
• Measurement never joins an inventory response. An unavailable store is a 503 with a reason.
• The plane makes outbound HTTP to the store at request time.
• ADR 0008 amends where the store is assumed to be and how queries are scoped. ADR 0011 and ADR 0012 move the per-cluster label and the write credentials onto the registry.

ADR 0004: One HTTP API for console and CLI

Decided: libs/packages/contract/API.md is the single document for the HTTP surface. The CLI and the console were built against shapes each had invented, and both were conformed to it.

Why: Until one document settled the disagreements, the two clients’ fakes were the only description of the API, which meant there was none. Each disagreement was resolved on the reasoning the object model already uses:

• Cluster phase is declared and reachability is observed, so they are separate fields.
• Collections share one list envelope, {items, count, next_cursor}.
• Ids use one prefix spelling contract-wide (clu_, lowercase Crockford body).
• api_endpoint and kubelet_version are dropped, because the plane never dials a cluster and only the operator knows Kubernetes exists.

Consequences:

• The CLI and the console are conformed to API.md, not the other way round.
• When the decision was made there was no automated conformance test for the HTTP surface, so agreement between the two clients and API.md was by inspection.

ADR 0005: The hub scales by shedding session state

Decided: SessionState (cursors and replay buffer) stays in memory. Horizontal scaling is achieved by making state loss correct and cheap, not by making it impossible.

Why: Sharing the state means writing the replay buffer to Postgres: up to 4096 frames per session, each written on send and deleted on ack, for every connected cluster continuously. That is a high-churn write load on the one database the entire control plane depends on, and it buys a property the protocol was explicitly designed not to need. full_resync is not a failure path: it’s the documented, correct answer to a lost cursor.

Shedding also degrades in the right direction. Under shared state the database is in the path of every frame, so database trouble becomes stream trouble for every cluster at once. Under shed state a hub pod dying costs exactly the clusters attached to it exactly one resync each, and the blast radius of a rollout is bounded by replica count, not by fleet size.

Consequences:

• The hub may run N replicas. Resume is best-effort and works within a pod; across pods a reconnect costs a full_resync, which is expensive but never wrong.
• Ownership of a cluster’s stream is durable: one hub session is the writer for a given cluster_id, claimed by an atomic compare-and-set at hello time. A displaced session discovers this by polling its own row and closes.
• Single-writer is enforced with a bounded overlap, not instantaneously. For up to one poll interval the displaced session may still write. It is by construction the older and staler of the two, so what that window can produce is a stale row and a flap of reachable, never a divergence that survives.

ADR 0006: Adoption is a decision to observe

Decided: Discovery and adoption produce no declared state at all. Tracking a workload means it appears in the inventory with a name and a project. It does not mean the plane manages it. The workload keeps running exactly as its own cluster has it configured.

Why: The operator is routinely pointed at clusters that are already serving: models running, weights resident in VRAM, and traffic on them, none of it declared by the plane. The obvious shape is an import: read the running Deployment, synthesize a workload, and let the reconciler take it from there. That shape restarts every model it touches. A declared object is something the cluster is obliged to match. The moment one exists, the operator has a difference between what is said and what is running, and closing that difference is a rollout.

Consequences:

• The operator reports workload.discovered: a sighting. Observed, full-object, on the supply stream, pruned like any other observation.
• Ingestion writes it to demand_discovered_workload and is that table’s only writer.
• POST /v1/discovered/{ref}/track writes one row in demand_workload_adoption: a decision, an owner, a note, who decided.
• stream_desired reads neither table. There is no path from a tracked workload to a deployment.desired.
• A tracked workload is visible but not controllable: it cannot be scaled, rolled, or upgraded through the plane, because the plane has declared nothing to change.
• Promotion (letting the plane manage a workload it adopted) is a separate, explicit act. See ADR 0009.

ADR 0007: Device identity comes from the DCGM exporter

Decided: Device identity comes from the DCGM exporter the cluster already runs, which the operator reads inside the cluster.

Why: Kubernetes does not know a GPU’s UUID; node labels say how many cards there are and what model, not which cards. Every capacity question that matters is per device: “seven of eight healthy” cannot be expressed as gpu_count: int. Which GPU is throttling, which has ECC errors, or which is running what are all per-device facts.

Vendor strings map to a Wingsmith device class through one table, libs/packages/contract/devices.json, read by both the operator and the plane.

Consequences:

• Identity precedence per device slot is the DCGM scrape, then the hangar.dev/device-uuids annotation, then the synthetic <node>-gpu<index> id.
• An unreachable exporter never fails discovery: it logs, and the node’s devices fall down that chain. A GPU cluster with no exporter still enrols and takes placement, but its device metrics do not join.
• Where the Kubernetes GPU feature labels and DCGM disagree on how many devices a node has, the feature-label count wins.
• The exporter is read inside the cluster by the operator. Only the identity reaches the plane, in device.observed.
• HANGAR_DCGM_ENDPOINT sets the exporter address. An empty value disables the scrape.

ADR 0008: Metrics are answered by a backend, not by PromQL

Decided: The metric vocabulary is vendor-neutral and a Backend answers metrics for a scope over a window. PromQL is one backend’s dialect, not the API.

Why: ADR 0003 assumed no shared store, so a query with no cluster in it could not blend clusters. The expected setup is now one central Prometheus that clusters remote-write into, stamping cluster and region on every series, and that store can hold clusters that are not inference clusters. An unscoped query then returns a plausible number that adds inference and database fleets together. The metric registry also carried the PromQL that answered each metric, so the vocabulary clients depend on was a Prometheus vocabulary. Run against a live store, this showed up as node-scoped panels returning no series because the exporter label was hostname, not Hostname, and as upstream renames of engine series (inter-token latency, KV cache usage) looking identical to an idle cluster.

Consequences:

• A Metric is a Wingsmith name, unit, family, title and aggregation. It names no series, label or query language.
• The dialect is a per-backend table keyed by the Wingsmith name. A second store is a second table.
• A Backend implements answers, samples, breakdown and curves over a Scope in Wingsmith’s dimensions. Adding ClickHouse or OTel is a new file, not a refactor.
• Every query carries a cluster matcher. The only exception is a source configured with its own endpoint and no label, which is a store holding one cluster.
• A metric lists candidate series in preference order, rendered as a PromQL or chain, so renamed series keep both spellings and a missing one gives a clean empty.
• A summed metric never unions its candidates. During an upgrade window an engine can emit both the old and new name, and a union would add the same tokens twice. Summed metrics probe in order and use the first candidate that has any series.
• A cluster with no metrics source still says so. It is not pointed at the shared store by default, because a guessed label returns a clean empty that reads as a healthy idle fleet.
• When a panel is empty, check the dialect table against a live store first.

Points of ADR 0003 stand: the plane queries and the browser never does, measurement never joins an inventory response, an unavailable store is a 503 with a reason, and absent is never zero. See ADR 0011 for where the per-cluster label now lives.

ADR 0009: Authority is a mark, not a consequence of adoption

Decided: Authority is a field, it is binary, and it is on the deployment: authority: observed | managed.

Why: ADR 0006 made adoption permanent: the object was adopted, so the operator refused to apply, always. The refusal is correct for a workload you’re only watching and wrong for one you’ve since decided the plane should run, and there was no way to say which was which. “Adopted” was doing two jobs: recording where an object came from, and deciding what the operator was allowed to do. Those change independently.

Consequences:

• observed: the plane records this workload and verifies it against what it was pinned to. It writes nothing to the cluster, ever.
• managed: the plane renders and converges it, exactly as it does for a deployment it created.
• The operator gates on authority, not on the presence of adopt. adopt stays and keeps its meaning (the live object’s identity and the digest it was pinned to), but it’s now provenance and a drift check instead of a prohibition.
• Observation is lossless and does not consult authority. The full observed spec is recorded for every sighting regardless of what you intend to do with it, alongside spec_fingerprint.
• Imported parameters are recorded separately from declared ones. The workload carries imported_args beside engine_args. The split is provenance, and provenance is what makes promotion reviewable.
• Managing a workload becomes possible, and stays deliberate. PUT /v1/deployments/{ref}/authority is the transition, in both directions.
• observed remains the default everywhere. Nothing acquires management by omission.

ADR 0010: Workload is the global umbrella

Decided: Workload becomes the parent of Deployment. IS-A becomes HAS-MANY.

Why: Deployment is workload × cluster: the unit the operator reconciles, one row per cluster. Workload existed above it as a supertype, joined 1:1 (Deployment was “Workload, specialized for serving”) which meant the two objects had the same cardinality and Workload carried no fact Deployment did not. The result is that nothing in the system expressed multi-cluster intent. Running one model in two regions is two independent Deployments that share nothing, plus an Endpoint whose target weights someone edits by hand to match.

Consequences:

• Deployment.workload_id survives and its unique constraint is dropped. That dropped constraint is the whole of the change at the schema level.
• No existing noun changes meaning. Deployment is still workload × cluster, still what deployment.desired carries, still what the operator reconciles.
• Facts sort by whether they’re global or per-cluster. Artifact and serving configuration, adapters, overrides, labels, kind, priority, preemptibility, gang size, deadline, topology requirement, and total scale are global and live on Workload. Cluster, admission phase and reason, timestamps, and this cluster’s share of the scale are per-cluster and live on Deployment.
• Scale is declared as a total; placement derives the shares. Workload carries total min/max/desired; placement derives each Deployment’s share as a fixed replica count. The default strategy is pack, not spread.
• Splittability is a property of kind. service and task split; gang does not: its ranks must be co-scheduled, so a gang workload has exactly one Deployment. Only service is accepted today.
• Scaling is an operation: workload.scale, workload.pause, workload.drain.

ADR 0011: A cluster carries its own metrics label

Decided: Cluster.metrics_label is declared state on the registry row: the cluster label value this cluster’s series are stamped with in the shared store. It is set at registration to the cluster’s own name.

Why: ADR 0008 kept the per-cluster label in a deployment variable, HANGAR_METRICS_SOURCES. That meant enrolling a cluster required a ConfigMap edit the register flow never mentioned. The chart already stamps series with the cluster name it is installed with, so asking for it a second time only creates a second thing that can disagree.

Consequences:

• Enrolling a cluster no longer touches the deployment. Register, install the chart, and the label is already the one the agent stamps.
• A null label still means no source, and the cluster is not queried under a guessed one.
• HANGAR_METRICS_SOURCES keeps the store endpoint and credential, which are process-level, not facts about a cluster.
• An explicit env entry still wins over the column. <cluster>|label=<name> overrides the label for one cluster, and <cluster>=<url> names a store holding that cluster alone, queried with no cluster matcher.
• A registered cluster that never enables monitoring.enabled now renders a dash instead of “no metrics source”. That is a weaker signal, bounded by the rule that a missing series is null end to end and never draws as 0%. Clear the label to get the explicit message back.
• Existing rows were not backfilled. A cluster can be left out on purpose because its series stopped landing, and a blanket backfill would turn that into an empty chart. Opting one in is an UPDATE or a PATCH.
• GET /v1/metrics/sources walks the registry instead of parsing the variable. Env entries that match no cluster are still listed, so a typo is visible.

ADR 0012: The plane is the source of the metrics write htpasswd

Decided: Each cluster’s enrolment token has two verifiers, and the plane renders the metrics ingress’s htpasswd file from the registry and patches the Secret that holds it.

Why: ADR 0011 moved the read side of a cluster’s metrics identity onto the registry row. The write side needed the same, or enrolling a cluster still meant editing credentials by hand. A separate telemetry credential would be a second thing to mint, store, rotate and revoke, and a second thing that can drift from the first.

Consequences:

• Registering a cluster stores a sha256 of the token (Cluster.token_hash), which the hub looks up by in constant time, and a bcrypt hash of the same plaintext (Cluster.remote_write_hash). The hub needs an indexed lookup, which bcrypt forbids, and Traefik can only check a crypt-style hash. Both are written together, so a cluster that rotates its token keeps its telemetry.
• A reactor pass renders <cluster-name>:<bcrypt-hash>, one per line sorted by name, and patches the Secret’s users key. It compares first, so the unchanged case writes nothing. It lives in the reactor because the reactor already elects one fleet-wide holder under a lease, and this is the one thing in the plane with an external side effect that needs exactly one writer.
• The plane talks to the Kubernetes API directly over httpx, with two verbs on one resource kind and a SecretStore protocol so it is testable without a cluster.
• It is gated on HANGAR_REMOTE_WRITE_AUTH_ENABLED and inert without an in-cluster service account, so a plane still runs on a laptop.
• The chart’s values.yaml carries no credential. The plane emits cluster.region, monitoring.enabled and monitoring.remoteWrite.url, and the chart derives the basic-auth credentials from the enrolment-token Secret it already creates.
• Clusters registered before this change have no remote_write_hash, because the plaintext exists only at mint. Their agents get a 401 until the enrolment token is re-issued and the chart reinstalled with the new values.
• Whatever populated the Secret before must stop, or two writers flap one key. If an external sync owns the Secret, replace it with an empty stub (users: "") instead of deleting it.
• The Role and RoleBinding that let the plane patch the Secret are applied once by hand, in the namespace the Secret lives in.

The one widening. The plane now writes a Kubernetes Secret. This does not breach the rule that only the operator talks to Kubernetes, because it is the plane’s own cluster and not a fleet cluster, it is one object by name with get and patch only (create is excluded because resourceNames cannot scope it), and it is a periodic diff-and-patch of one key, not a controller.

Limitations:

• Per-cluster credentials give revocation granularity, not label integrity. Traefik authenticates the connection and does not police external_labels, so cluster A can write series labelled as cluster B. Do not treat the cluster label as an authenticated claim. Enforcing it would need a write proxy in the data plane that rewrites labels from the authenticated identity.
• Bcrypt on the ingress costs CPU on every remote-write request, and on a public endpoint that is a denial-of-service lever. The credential is secrets.token_urlsafe(32), so the slow hash protects against an attack that does not apply, and the cost factor is 8 instead of the library default of 12. A rate limit in front of the middleware is not built.

ADR 0013: One frontend per model, routed by a forwarded deployment id

Decided: The operator renders a frontend for each model as part of that model’s deployment. One frontend per model. The gateway forwards the deployment id it already resolved, as X-Hangar-Deployment-Id. The agent maps that id to a Service by the hangar.dev/deployment-id label.

Why: A cluster ran exactly one OpenAI-compatible Dynamo frontend and served every model through it. The frontend is a binary, and it must be able to tokenize and register every model it discovers. So the frontend image was a fleet-wide lowest common denominator: every model on a cluster had to be servable by one runtime build. That constraint breaks the first time a model needs a frontend fix the shared build does not have.

Consequences:

• Cluster selection stays global and device selection local. The global decision (which cluster, which deployment) stays the plane’s, and it is not re-decided in the cluster.
• The frontend is declared state, not an operation. It is rendered from the workload like everything else the operator renders.
• A frontend is not a replica. It’s a separate component.
• A frontend that is not serving degrades its deployment. Under one frontend per model it is the only way in, so a complete set of ready workers behind a frontend that never started is not running.
• Pod count grows with models, not with image diversity. Two frontend pods per deployment where a cluster ran two in total.

ADR 0014: Recipe is deleted; serving config lives on the workload

Decided: Recipe is deleted as an object. demand_recipe is dropped and its columns move onto demand_workload with the same names and types. The serving configuration reaches a cluster inlined on deployment.desired.serving.

Why: Recipe was a declared object of its own, with a name, version, lifecycle, row, API surface and wire frame. Two claims justified that: a recipe is shared by many workloads, and it travels once to a cluster that many deployments then refer to. Neither held up in the code.

• Cross-cluster sharing saved nothing. The demand stream was built per cluster, so a configuration used in three clusters was serialised three times, exactly as three inlined copies would have been. The only saving was two deployments in one cluster naming one recipe, which is the canary case.
• The separate frame had a price: an ordering rule on the demand stream, a RecipeNotDeclared pending reason, and a resume race in which a reconnecting operator got a deployment but not the recipe it named.
• The sharing it offered was being reconstructed instead of used. The console compared eight fields of a candidate against every existing recipe on the artifact to decide whether one was the same. Comparing contents to find an object means the contents are the object.

Consequences:

• /recipes and its sub-resources are gone from the HTTP API, hgr recipe* from the CLI, and the recipe routes from the console.
• The serving configuration reaches a cluster inlined on deployment.desired.serving. recipe.desired, the ordering rule, and RecipeNotDeclared are all removed with it.
• Reuse is serving_digest. A svg_-prefixed sha256 over canonical JSON of exactly the fields that decide what process runs. Two workloads with the same configuration have the same digest, and that is the whole mechanism.
• The three-level split becomes two, and the argument for it survives intact. Llama-70B at FP8/TP2/32k and the same weights at BF16/TP4/128k are still different hardware footprints, still different price points, still plausibly different clusters. They are now two Workloads over one Artifact, told apart by their serving_digest.
• Benchmarks anchor on something that cannot drift. A run now records serving_digest. A digest that no longer matches any workload is a configuration nobody runs anymore, which is a true statement; a recipe_id that still resolves to a row someone has since edited is a false one.
• The real cost: Recipe.name, Recipe.version, and Recipe.lifecycle are gone and are not replaced. A workload has a name; a serving configuration no longer has one of its own.

ADR 0015: The frontend router is declared on the workload

Decided: serving.router is a typed block of four optional fields (mode, queue_threshold, queue_policy, kv_overlap_score_credit), each rendered as exactly one DYN_ROUTER_* variable on the deployment’s own frontend. It is stored in its own column, demand_workload.router.

Why: The operator rendered one router setting, DYN_ROUTER_MODE=kv, unconditionally. A large MoE model can need three more settings to hold its latency targets under load. With nowhere to declare them, the only way to run them was a hand patch on the graph deployment with the operator scaled to zero, which is a cluster the plane no longer controls.

Consequences:

• The block is separate from serving because serving replaces whole on a patch, and a router setting that vanished when someone edited a parser would silently undo a benchmark configuration.
• Undeclared renders what rendered before: no block, or all nulls, is DYN_ROUTER_MODE=kv and nothing else. No frontend rolls when the plane or operator learns the field.
• The block is part of serving_digest only while something in it is set, so existing digests are unchanged.
• Values are closed on both sides of the wire: mode is the frontend’s router modes except direct, queue_policy is fcfs | wspt, and numbers are finite and zero or more. A policy without a threshold, or a KV setting beside a non-KV mode, is refused because neither takes effect.
• There is no environment passthrough. The next setting that matters becomes a field with its own validation.
• Setting a router is a serving change: it bumps the generation and digest, and the operator applies the graph once. The block reaches the frontend’s pod template only, so no worker is rolled, which matters where a worker takes hours to load.
• The plane does not know which Dynamo version a frontend runs. A mode the pinned version accepts may not exist in an older frontend image, which would exit at startup, so the set has to be revisited when the fleet’s frontends change.
• Server-side apply keeps fields the render does not produce. Declaring a router does not clean up a hand patch it replaces.

ADR 0016: The listener is the guard; tenant identity replaces the shared token

Decided: Which socket accepted the connection is the trust primitive, not a header any caller can send. Tenant identity comes from Host, checked only where the zone requires proof. Guard, TokenGuard, and HANGAR_GATEWAY_TOKEN are deleted.

Why: The gateway’s own README used to say “no authentication, metering, or billing,” and Guard’s doc comment said “this is not user authentication and must not grow into it.” Both were written when the gateway had exactly one caller (the operator’s own inference edge, which authenticates its users before calling here), and the only question worth asking was whether a request came from that caller at all. HANGAR_GATEWAY_TOKEN answered it: a shared secret, checked in constant time, with no notion of who was on the other end beyond “holds the current token.”

A second tenant changes the question. A tenant that calls the gateway directly means the gateway now has to know which tenant a request belongs to, not only whether it is allowed to ask at all. A shared secret cannot answer that.

Consequences:

• internal/serving.Zone is fixed at listener construction: ZoneInternal for the existing ClusterIP-with-no-ingress port, ZonePublic for a new listener behind a wildcard.
• On the internal zone, reaching the port is the proof and a tenant may declare auth: none.
• On the public zone, the Host header resolves a tenant and, when that tenant declares auth: api_key, a presented key’s hash must belong to that tenant specifically.
• The public listener is off until HANGAR_GATEWAY_PUBLIC_LISTEN names an address.
• Routing is scoped by tenant on the public zone only, and a stranger’s model is 404. Resolving a name that belongs to another tenant is treated exactly like resolving a name that does not exist.
• An existing edge needs no change at all. It stays internal + none, dialling the same in-cluster hostname it always has.

ADR 0017: Video input is bounded in decoded bytes, not frames

Decided: A video request is bounded by the bytes it decodes to, using a bound that is flat in clip duration and source resolution, instead of by a frame cap.

Why: A video_url request could exhaust a worker’s host memory. The video loader preallocates one array for the whole clip at native resolution, and the runtime image set the frame count to all frames. Five minutes of 1080p30 is about 52 GiB and twenty minutes of 4K is about 834 GiB, multiplied by the loader thread count. The clip arrives as a URL, so the gateway’s body cap never applies. A frame cap bounds one factor of frames x height x width x 3, and nothing in the stack bounded the other usefully. Across providers, bytes and duration are rejected and pixels are silently downscaled.

Consequences:

• The admitted clip length is twenty minutes, matching a comparable provider. Bound 2 is what makes that safe, since bound 1 alone leaves a native 1080p twenty-minute clip near 60 GiB.
• Bound 5 is required: media_io_kwargs is a per-request field, so without it bound 1 is one request away from being defeated.
• The gateway cannot bound the duration or bytes of an https:// clip, because fetching a caller-supplied address is an egress decision. Bounds 3 and 4 live in the worker, where the fetch happens. Bound 4 does not cover data: URLs, which are limited only by the gateway’s body cap when they cross it.
• Measurement corrected the original figures. The algebraic ceiling of 367.5 MiB assumed an exact resize, and the measured worst case is 723.4 MiB. The shape of the bound is unchanged.
• Twenty minutes needs the encoder admission gate raised, which otherwise rejects clips above roughly a minute at 1080p.
• A clip longer than the patch budget can resolve gives the model less detail per second. This is documented, not hidden.
• Bounds 2, 3 and 4 are a patch to the worker’s video loader, carried in a worker image layered on the vendor runtime. Bound 1 has one route: a flag reaches a worker only through that workload’s own engine_args.

ADR 0018: The inference tunnel is HTTP/2 role reversal

Decided: Wingsmith carries its own tunnel, in libs/packages/tunnel, built on HTTP/2 with the roles reversed. The agent dials the gateway over TLS, presents its enrolment token in a single length-prefixed handshake (HGRTUN01), and then runs the HTTP/2 server on that connection while the gateway runs the HTTP/2 client.

Why: Nothing may dial into a GPU cluster, but the gateway has to send requests to a model’s frontend inside one. Several mature reverse tunnels exist: frp, chisel, cloudflared, ngrok, Konnectivity. What the gateway needs from the tunnel is narrower than “reach a port in the cluster”: an http.RoundTripper per cluster, retry of unprocessed requests on a sibling connection, a fast answer when a cluster is full, identity as the cluster’s enrolment token, coverage of every gateway replica from one address, and no extra process or vendor in the request path.

Consequences:

• The module depends on golang.org/x/net/http2 and nothing else, and implements both ends, because neither the gateway nor the agent owns the protocol.
• The pool implements http2.ClientConnPool, so http2.Transport’s own retry runs and the pool hands it a different live tunnel.
• The response-header timeout is the upstream one, armed at request write and disarmed at the first header.
• A full cluster returns ErrSaturated at once instead of blocking for a stream slot.
• The handshake reply names the gateway replica and how many there are, so the agent redials one address until it holds a tunnel to each.
• The protocol is versioned by its magic. A wire change is a new magic, so an agent speaking the wrong version is rejected before anything is misread.
• The threat model is part of ADR 0018 and changes with the code. Today: TLS on the tunnel listener, a 10 s handshake deadline, an 8 KiB handshake cap, tokens verified by the plane and per cluster. Verifications are cached for 1 minute, and for up to 15 minutes while the plane is unreachable, so a revoked token can open new tunnels for that long.