Maintain

Upgrades

Roll a built-from-source Wingsmith install forward: migrations, plane, gateway, then clusters.

Wingsmith is pre-release. There are no published images or chart, no tagged releases and no supported upgrade path between versions. This page is the order that works when you build every image yourself and move from one build to the next. Expect to read the commit log for anything breaking.

Order

Plane first, clusters after. The hub is the side that carries compatibility code, so it must move before the operators.

Back up Postgres

Migrations can drop or rewrite data and are not guaranteed to be reversible in practice. The database is the only copy of the plane’s state. Restore from this backup if a migration fails partway or you need data back after a downgrade.

Build and push the new images

Use a new tag. See Build images.

export HANGAR_REGISTRY=registry.example.com/hangar
just push plane <tag>
just push gateway <tag>
just push operator <tag>
just push agent <tag>
HANGAR_CHART_OCI=oci://registry.example.com/hangar/charts just chart-push

just push refuses a tag the registry already has, so every build needs a new one.

Run migrations

Run migrate upgrade head with the new plane image and wait for it to finish, before rolling anything.

kubectl run hangar-migrate --image=$HANGAR_REGISTRY/plane:<tag> \
  --env=HANGAR_DATABASE_URL=<database url> \
  --restart=Never --rm -it -- migrate upgrade head

Or use your deployment tool’s one-shot job. It needs the same HANGAR_DATABASE_URL as the api.

Roll the plane, then the gateway

Roll the api, hub, and reactor if you run it separately, on the new image. Then roll the gateway.

Point clusters at the new images

On the plane, set HANGAR_OPERATOR_IMAGE, HANGAR_AGENT_IMAGE and HANGAR_SETUP_CHART_VERSION. The plane writes them into each cluster’s setup values, so regenerate values for every cluster you have not upgraded.

Upgrade each cluster

Apply changed CRDs first if the change touches them (see CRDs), then:

helm upgrade --install hangar-cluster <chart ref> \
  --version <chart version> \
  --namespace hangar-system \
  -f values.<cluster>.local.yaml

Use the chart ref, version and values file your plane renders into the cluster’s setup instructions. To upgrade from a checkout instead, run helm dependency build libs/chart/cluster-setup and pass that path as the chart. The operator and agent image tags default to the chart’s appVersion, so keep the image tags and the chart build together.

Migrations

Migrations are Alembic, with one linear chain and one head. They need Postgres 15 or later.

Two runs started together apply each revision once. The second waits on an advisory lock, then finds nothing to do.
All revisions in one run share a transaction, so a failure leaves the database at the revision it started from.
Revisions that rewrite tables hold an exclusive lock while they copy. A revision that widens a sequence column rewrites every table, so on a large operations table expect it to take minutes.
Some revisions refuse to run instead of guessing. One lists workloads with no deployment and stops until you give them a deployment or pass alembic -x drop_orphan_workloads=true upgrade head. Another lists duplicate fleet-wide quotas or credentials and stops until you delete all but one of each. Read the output of a failed run.
One revision deletes policy attachments whose tenant, project, cluster, endpoint or workload no longer exists. They point at nothing.

The window between migrate and roll

Some revisions rename or drop columns in place. Between the migration finishing and the plane pods rolling, old api and hub pods may fail requests that touch those columns. Running new code against the old schema is also unsupported. Either accept a short window of plane API errors, or scale the api and hub to zero during the migration. Roll immediately after migrating.

This window affects the plane’s API and hub only. Clusters keep running from local state and the gateway keeps serving from its cached table. See Plane outages.

Downgrades

Every revision has a downgrade, but a downgrade restores the schema and not always the data. Where an upgrade discarded information, the downgrade rebuilds the old shape from what is left. Treat the backup as the way back.

Version checks

Two checks happen when an operator connects to the hub.

Check	Rule	On failure
Release line	An operator reporting a release-shaped version (`0.4.2`, `v0.4.2`) outside the major the plane speaks is refused. A non-release version such as `dev` or a commit hash is a build from source and is admitted with a log line	Protocol error close, not retryable
Envelope version	Each side writes one envelope version `v` and reads a range from a minimum to its current. A frame outside the range closes the connection	The hub answers with a non-retryable `error`. The operator drops the session with an error naming the range it reads

Today both sides write and read envelope version 1, so the second check cannot fail between two builds from this tree. It matters when a future change widens the range: that is rolled out hub first, then operators, then the hub’s writer, then the floor is raised.

There is no negotiation on hello. Unknown frame types and unknown fields do not break a session: the receiver drops the frame and answers with an error, which is how an operator one build behind the plane keeps running. State that never converges after an upgrade is the symptom to look for, so check operator logs for these errors.

Because source builds are admitted regardless of version, the checks do not stop you running a plane and operator from builds far apart. Move clusters soon after the plane.

CRDs

Both CRDs, SyncedResource and StreamCheckpoint, are v1alpha1 in the hangar.dev group, shipped in the chart’s crds/ directory. Helm installs them once and never updates them. A helm upgrade does not apply a changed CRD, install a new one, or remove one on uninstall.

When a chart change touches crds/, apply them yourself before the upgrade:

kubectl apply --server-side -f libs/chart/cluster-setup/crds/

The chart’s CHANGELOG.md lists changes that need this and any breaking value changes. Read it before upgrading a cluster, and note that below 1.0 a minor version may break values or CRDs.

Gateway rollout

The gateway sits in the inference request path, so roll it with that in mind.

Run more than one replica, spread across zones, with a PodDisruptionBudget so at least one stays up.
Do not roll it while the plane is down: a new replica has no routing table and refuses to start.
Raise terminationGracePeriodSeconds and HANGAR_GATEWAY_SHUTDOWN_GRACE together if you serve long generations. The default is 30 s, and the drain phases are cut from it. On SIGTERM the replica reports not ready, drains in-flight requests, then drains the tunnel listener.
Change HANGAR_GATEWAY_REPLICAS and the replica count in the same rollout. Agents read the replica count from each replica’s handshake and redial until they hold a tunnel to every one. A value that is too low leaves replicas with no tunnel to some clusters, answering 503 no_capacity while reporting healthy.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: hangar-gateway
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: hangar-gateway

Adjust the selector to the labels on your gateway pods.

Hub and operator rollout

You can roll the hub without coordination. A reconnect that lands on a different hub pod costs that cluster a full resync instead of a resume. Resync is correct but more expensive, and a rolling restart of every hub pod resyncs every cluster.

Clusters keep serving while an operator or agent rolls. The operator reconciles from the declared state held in the SyncedResource objects and does not need the plane to do so.