Configuration
Operator Configuration HANGAR_* variables for operator: hub connection, namespaces, read-only mode, DCGM, pools, fabric.
The operator is configured entirely through environment variables. Three are required: cluster ID, hub URL, and enrolment token.
Variable Description HANGAR_CLUSTER_IDThe plane-assigned cluster id HANGAR_HUB_URLwss:// hub address; ws:// only with HANGAR_HUB_ALLOW_INSECURE=trueHANGAR_ENROLMENT_TOKENThe cluster’s enrolment token
Variable Default Description HANGAR_HUB_ALLOW_INSECUREfalseAccept a ws:// hub URL, which sends the token in the clear; local development only HANGAR_HUB_CA_FILEnone PEM bundle for verifying the hub HANGAR_HUB_INSECURE_SKIP_VERIFYfalseSkip TLS verification; local development only, logs a warning at startup HANGAR_HEARTBEAT_INTERVAL10sApplication-level ping interval HANGAR_LIVENESS_TIMEOUT30sSilence tolerated before redialling; must exceed the interval HANGAR_BACKOFF_MIN / HANGAR_BACKOFF_MAX1s / 30sReconnect backoff, full jitter
Variable Default Description HANGAR_NAMESPACEhangar-systemWhere the custom resources and the lease live HANGAR_WORKLOAD_NAMESPACEdynamoWhere rendered workloads and prefetch jobs go
Variable Default Description HANGAR_READ_ONLYfalseObserve and report only: no reconciler runs and operations are refused HANGAR_ALLOWED_OPERATIONSnone Operation kinds a read-only operator still runs, comma-separated
A read-only operator has no reconciler to tear anything down. It runs one small controller that removes the finalizer from any declaration being deleted and leaves anything rendered in place.
On a managed cluster all three operation types (cordon, drain, benchmark) are allowed. On a read-only cluster they come from HANGAR_ALLOWED_OPERATIONS (empty by default, so none).
Variable Default Description HANGAR_DCGM_ENDPOINTin-cluster exporter Where device identity is scraped HANGAR_DCGM_SELECTORnone Pod label selector for DCGM exporters HANGAR_DCGM_TIMEOUT5sScrape timeout
The operator scrapes DCGM’s Prometheus metrics for device identity (UUIDs), device classes, topology domains, and health.
Variable Default Description HANGAR_POOL_LABEL_KEYhangar.dev/poolNode label workers are placed by HANGAR_GPU_POOL_VALUEgpuValue for the GPU pool HANGAR_CACHE_POOL_VALUEcacheValue for the weight download pool
Variable Default Description HANGAR_GPU_TOLERATION_KEYnvidia.com/gpuThe taint GPU workers tolerate HANGAR_GPU_TOLERATION_VALUEnone Value for the GPU toleration; no value tolerates any value of the key HANGAR_CACHE_TAINT_KEYhangar.dev/cacheTaint key for weight download nodes HANGAR_CACHE_TAINT_VALUEnone Value for the cache taint
Variable Default Description HANGAR_MODEL_CACHE_PVCmodel-cacheWhere weights are cached HANGAR_LOCAL_MODEL_CACHE_DIRnone Local model cache directory HANGAR_PREFETCH_CPUnone Prefetch job CPU request HANGAR_PREFETCH_MEMORYnone Prefetch job memory request HANGAR_PREFETCH_MEMORY_LIMITnone Prefetch job memory limit
The operator prefetches model weights with a Job before any worker starts. Weights are cached on a ReadWriteMany PVC mounted by name.
Variable Default Description HANGAR_ENGINE_PULL_SECRETSnone Image pull secrets for engine workers HANGAR_ALLOWED_IMAGE_PREFIXESnone Registries or repositories every engine, frontend, and prefetch image must come from; empty allows any
HANGAR_ALLOWED_IMAGE_PREFIXES is a security boundary. When set, the operator refuses any image that does not start with one of the listed prefixes before creating an object.
Variable Default Description HANGAR_FRONTEND_ENABLEDfalseRender one frontend per model HANGAR_FRONTEND_IMAGEnone The frontend’s image HANGAR_FRONTEND_REPLICAS2Frontend replica count HANGAR_FRONTEND_PULL_SECRETSnone Frontend image pull secrets HANGAR_FRONTEND_CPUnone Frontend CPU request HANGAR_FRONTEND_MEMORYnone Frontend memory request HANGAR_FRONTEND_MEMORY_LIMITnone Frontend memory limit HANGAR_FRONTEND_ANTHROPIC_APItrueServe /v1/messages HANGAR_FRONTEND_ANTHROPIC_STRIP_PREAMBLEtrueDrop a fixed client preamble so the prefix cache can hit HANGAR_FRONTEND_GLOBAL_DYNAMO_NAMESPACEtruePut each graph’s frontend and workers in the shared Dynamo namespace; false gives each graph its own
Read only with HANGAR_FRONTEND_ENABLED. When true, the operator renders a Dynamo frontend into each model’s own graph.
Variable Default Description HANGAR_MULTINODEfalseAssert a leader-worker backend (Grove or LeaderWorkerSet) when discovery cannot see it HANGAR_FABRICnone RDMA fabric workers move KV over: empty, efa, or rdma HANGAR_FABRIC_DEVICES0 (except efa)Interfaces per worker pod HANGAR_FABRIC_RESOURCEvpc.amazonaws.com/efa under efaExtended resource for the fabric HANGAR_FABRIC_PRIVILEGEDtrue under efa, else falseRun fabric workers privileged with host IPC HANGAR_EFA_DEVICESnone Legacy alias for HANGAR_FABRIC=efa with that many devices; refused alongside the new keys HANGAR_NIXL_BACKENDSnone NIXL plugins the KV connector registers; LIBFABRIC on EFA
For AWS EFA:
HANGAR_FABRIC = efa HANGAR_FABRIC_DEVICES = 4 HANGAR_FABRIC_RESOURCE = vpc.amazonaws.com/efa HANGAR_FABRIC_PRIVILEGED = true For RDMA (requires setting the resource):
HANGAR_FABRIC = rdma HANGAR_FABRIC_DEVICES = 2 HANGAR_FABRIC_RESOURCE = rdma/hca Variable Default Description HANGAR_INFERENCE_ENDPOINTdiscovered The frontend a benchmark aims at HANGAR_BENCHMARK_TIMEOUT30mBenchmark run timeout HANGAR_BENCHMARK_CPU500mBenchmark job CPU request HANGAR_BENCHMARK_MEMORY1GiBenchmark job memory request HANGAR_BENCHMARK_IMAGESpinned aiperf and evalscope images Harness images a benchmark may run, matched exactly HANGAR_BENCHMARK_TOKEN_SECRETnone Secret in the workload namespace handed to benchmark jobs as environment, for a gated tokenizer HANGAR_BENCHMARK_ARTIFACT_BASEnone Where raw benchmark records would be uploaded; nothing uploads yet
The operator runs benchmarks the plane sends as Jobs in the workload namespace. HANGAR_BENCHMARK_IMAGES is a security boundary: the operator refuses any harness image not on the list.
Variable Default Description HANGAR_OBSERVE_INTERVAL30sObservation tick HANGAR_OBSERVE_TIMEOUT60sObservation deadline HANGAR_OBSERVE_STALE_AFTER5mReadiness fails when the last observation is older HANGAR_RESYNC_INTERVAL15mHow often the whole observed inventory is restated, which retires objects the cluster no longer has HANGAR_SEND_QUEUE_SIZE1024Outbound queue bound
Variable Default Description HANGAR_LEADER_ELECTIONtrueController-runtime leader election HANGAR_HEALTH_ADDR:8081/healthz, /readyzHANGAR_METRICS_ADDR0Controller-runtime metrics; 0 disables
Run one replica. Leader election is on by default and guards against two operators running at once during a rollout. A replica waiting for the lease reports not ready because it holds no hub session.
Liveness is unconditional. Readiness carries the hub session and the observation loop: a partitioned operator is alive, because the cluster is meant to keep working when the plane is unreachable.
Variable Default Description HANGAR_LOG_LEVELinfoLog level HANGAR_LOG_FORMATjsonLog format
The operator renders for NVIDIA GPUs only: workers request nvidia.com/gpu, and device identity comes from DCGM. Other accelerators are not supported.
