The cluster-setup chart

What the cluster-setup chart installs in a GPU cluster, and how to configure, upgrade and remove it.

The cluster-setup chart turns a Kubernetes cluster with schedulable NVIDIA GPUs into one the plane can manage. To install it, follow Enrol a cluster. This page explains what it contains and the decisions behind its settings. Every key is listed in Chart values.

What it installs

One Helm release, by default named hangar-cluster in hangar-system.

The operator needs a ReadWriteMany storage class for the model cache (modelCache.storageClass) and a node label scheme that names a GPU pool and a cache pool (pools.*). The chart refuses to render without cluster.id, cluster.hubUrl, enrolmentToken.value, pools.labelKey, pools.gpuValue, pools.cacheValue and modelCache.storageClass.

Frontend topology

frontend.mode decides where a request lands inside the cluster.

The first install of a new cluster must use none, because a frontend is a DynamoGraphDeployment and the Dynamo operator registers that CRD only at runtime.

frontend.enabled and operator.frontend.enabled were removed. The chart fails with migration instructions if either is set.

Serving agent

Nothing dials into a GPU cluster. The agent is its own Deployment, separate from the operator, that dials out to the gateway and serves HTTP/2 on those connections while the gateway drives the client end. Requests arrive down a connection the cluster opened and are proxied to the local Dynamo frontend. The operator and the agent share neither a process nor a connection, so plane availability is not part of the inference path.

cluster:
  readOnly: false
agent:
  enabled: true
  replicas: 2
  gateway:
    address: gateway.example.com:443
  frontendEndpoint: ""   # empty finds the Dynamo frontend Service by label

agent.gateway.address is one host:port, however many gateway replicas exist. TLS is assumed. For a gateway certificate from a private CA, set agent.gateway.caBundle. If the certificate name differs from the host, set agent.gateway.serverName. The agent presents the cluster’s enrolment token as its identity, so there is no second credential.

The chart refuses these instead of installing something that looks healthy and carries nothing:

• agent.enabled with cluster.readOnly: true. A read-only cluster applies nothing, so the agent would hold tunnels and answer every request with a 502.
• agent.enabled with frontend.mode: none and no agent.frontendEndpoint.
• agent.enabled with an empty agent.gateway.address, a comma-separated list, or an address with a scheme.
• an empty enrolmentToken.value.

The agent forwards only inference POST paths and answers anything else 404 unsupported_path. Its settings are in Agent configuration. How the gateway finds every replica from one address is in Deploy the gateway.

Hardware

The operator renders for NVIDIA GPUs only. Workers request nvidia.com/gpu and device identity comes from DCGM.

Telemetry

With monitoring.enabled, a Prometheus in agent mode scrapes DCGM and the Dynamo pods and pushes the series to monitoring.remoteWrite.url. cluster.region is required with it. Any store that accepts Prometheus remote write works, as long as the plane reads from the same store. Every series carries cluster (equal to cluster.id) and region labels.

Setting up the store and the plane side is in Monitoring.

Dynamo platform

The dynamo-platform subchart is NVIDIA’s chart, pinned at 1.4.1, and is on by default. The operator renders every model as a DynamoGraphDeployment, so nothing serves without it. It also brings NATS, which Dynamo’s frontends and workers use.

NATS has no client authentication here. The hangar-nats NetworkPolicy keeps other namespaces off it, so leave networkPolicy on unless your CNI does not enforce policies.

If Dynamo is already installed, or you want to install it yourself, set dynamo-platform.enabled: false and install a 1.4.x release separately. The Wingsmith operator needs only the DynamoGraphDeployment CRD and a Dynamo operator reconciling it.

A model’s engine runtime image and a per-model frontend image must come from the same Dynamo release as the platform, or workers and frontend disagree on the wire.

Operator permissions

The operator’s RBAC is cut to what the install allows, so cluster.readOnly is a permission boundary and not only a setting the operator honours.

readOnly defaults to true in the chart. The values file the console renders sets it to false. On a read-only cluster, cluster.allowedOperations lists the operations the plane may still request (cordon, drain, benchmark). It is empty by default. On a managed cluster all three are on.

On a managed cluster the plane decides what runs in the workload namespace, including engine and benchmark images, so whoever has plane admin access can run code there. Two values narrow that to images you chose, and the operator refuses anything else before it creates an object:

operator:
  images:
    allowedPrefixes: []   # empty allows any image; set e.g. ["registry.example.com/"]
  benchmark:
    images: []            # empty uses the plane's pinned harness images

Without an allowlist, treat plane admin as write access to the workload namespace.

Serving egress

Egress from the workload namespace is open by default. Media URLs in a request are fetched from inside the serving pods, so on an open network they can reach internal addresses, including a cloud metadata endpoint. The gateway refuses private and internal hosts first. networkPolicy.servingEgress.enabled closes the pods’ egress as well. They then reach only their own namespace, the release namespace, DNS, the API server CIDRs in networkPolicy.servingEgress.apiServer and the peers in networkPolicy.servingEgress.to. Check what your models fetch at startup before turning it on.

CRDs

The chart ships two CRDs in crds/, both in the hangar.dev API group at v1alpha1:

• SyncedResource (short name syncres): the operator’s local copy of what the plane declared
• StreamCheckpoint (short name streamcp): its resume cursor on the hub stream

They are the operator’s own state, not an API for you to write to. The schema can change between builds without a conversion webhook. Helm installs crds/ on first install and never touches it again, so helm upgrade does not update a changed CRD and helm uninstall does not remove it.

Versions

The chart’s appVersion equals its version. The operator and agent images default to that tag (operator.image.tag and agent.image.tag empty), so one chart build runs one build of each. Set a tag only to run a build of your own. operator.image.digest and agent.image.digest pin an exact build and take precedence.

Upgrade the chart

Run the plane and the chart from the same build. The order for the whole system is in Upgrades.

kubectl apply --server-side -f crds/

Pre-release, no compatibility between mixed builds is promised.

Uninstall

A declared deployment carries the hangar.dev/rendered finalizer until the operator has removed what it rendered. Uninstall in this order.

kubectl -n hangar-system delete syncedresources --all --wait

helm uninstall hangar-cluster -n hangar-system

kubectl delete -f crds/

If the operator is already gone, remove the finalizers by hand, then delete the graph deployments it rendered. They carry the hangar.dev/deployment-id label.

kubectl -n hangar-system get syncedresources -o name \
  | xargs -r -n1 kubectl -n hangar-system patch --type merge -p '{"metadata":{"finalizers":null}}'

A read-only operator has no reconciler to tear anything down. Delete anything an earlier read-write run rendered by that label.