Maintain

Monitoring

Scrape the Wingsmith components, set up the fleet metrics write path, and connect a metrics store to the console.

There are two separate things to set up. Your own Prometheus scrapes the plane, gateway, operator and agent for operational health. Separately, each cluster pushes GPU and serving metrics to a store the plane reads back for the console. Series names are in Metrics reference.

Scrape the components

Component	Port and path	Scrape from
Plane api	`:8000/metrics`	Every api pod, on the cluster network
Gateway	`:9090/metrics`	Every gateway replica
Operator	`:8080/metrics`	Every cluster, with `operator.metrics.podMonitor.enabled` or your own config
Agent	`:8081/metrics`	Every cluster

Scrape pods, not Services. The plane and gateway report only their own process, so a Service scrape returns a random replica each time.

/metrics on the api needs no credential. Refuse it at the ingress that serves the plane’s HTTP path and scrape the pods directly.

A scrape config for the plane and gateway, assuming they run in hangar-system with an app label (adjust to your manifests):

scrape_configs:
- job_name: hangar-plane-api
  kubernetes_sd_configs:
  - role: pod
    namespaces:
      names: [hangar-system]
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app]
    action: keep
    regex: hangar-api
  - source_labels: [__meta_kubernetes_pod_ip]
    target_label: __address__
    replacement: $1:8000

- job_name: hangar-gateway
  kubernetes_sd_configs:
  - role: pod
    namespaces:
      names: [hangar-system]
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app]
    action: keep
    regex: hangar-gateway
  - source_labels: [__meta_kubernetes_pod_ip]
    target_label: __address__
    replacement: $1:9090

The hub and reactor export no metrics. The cluster-side bundled metrics agent scrapes DCGM and Dynamo only, not the operator or the Wingsmith agent.

What to alert on

Signal	Series	Meaning
Plane unreachable from the gateway	`hangar_gateway_routing_table_age_seconds` rising with `hangar_gateway_routing_table_refresh_success` at 0	The gateway is serving a stale table. See Plane outages
A replica cannot reach a cluster	`hangar_gateway_tunnel_coverage == 0`	That replica answers `503 no_capacity` for the cluster’s models while still reporting ready
Tunnel certificate near expiry	`hangar_gateway_tunnel_certificate_expiry_timestamp_seconds`	Every agent dial fails once it passes
Incomplete tunnel coverage	Count of `hangar_agent_gateway_up == 1` below `hangar_agent_gateways_expected`	The agent is missing a tunnel to a replica
Failures at the gateway	`hangar_gateway_requests_total` by `outcome`	Exclude `client_error`, `auth_rejected`, `rate_limited` and `canceled` as your objective requires

Fleet metrics write path

The console shows GPU and serving charts by querying a metrics store through the plane. Clusters write into that store with Prometheus remote_write. Without a store the console shows inventory only.

Pick a store

Any store that accepts remote_write and answers the Prometheus query API: Prometheus with --web.enable-remote-write-receiver, Mimir or VictoriaMetrics. It is a separate deployment from the plane.

Tell the plane where it is

Set these on the api:

env:
- name: HANGAR_METRICS_REMOTE_WRITE_URL
  value: https://metrics.example.com/api/v1/write
- name: HANGAR_METRICS_SOURCES
  value: default=https://metrics.example.com

The plane writes HANGAR_METRICS_REMOTE_WRITE_URL into each cluster’s setup values and turns on that cluster’s metrics agent. HANGAR_METRICS_SOURCES is where the plane reads back: default=<base url of the query API>. For a store that needs a bearer token, append |token=<token>. The token is never returned to clients.

Set up write authentication

See the next section.

Register clusters

When you register a cluster, the plane records the cluster label its series will carry (the cluster’s name). The chart stamps cluster from cluster.id and region from cluster.region on every series, so register and install with the same name. No change to the plane’s configuration is needed per cluster.

A cluster whose monitoring.enabled is false pushes nothing, and its console charts show a dash instead of zero.

Cluster entries in HANGAR_METRICS_SOURCES

You rarely need per-cluster entries. Two forms exist as overrides, and an entry wins over the registered label:

Entry	Effect
`<cluster>\|label=<name>`	Query the shared store with this `cluster` label for one cluster
`<cluster>=<url>`	A separate store that holds only that cluster, queried with no `cluster` matcher

Entries are comma-separated and match a cluster by name or uid.

Authenticate remote write

The cluster’s metrics agent authenticates in one of three ways, set in the chart’s monitoring.remoteWrite:

Mode	Values	Sends
Enrolment token (default)	`basicAuth.useEnrolmentToken: true`	Basic auth, username `cluster.id`, password the cluster’s enrolment token
Separate credentials	`useEnrolmentToken: false` with `basicAuth.username` and `password`, or `basicAuth.existing: true` with `existingSecret`, `usernameKey`, `passwordKey`	Basic auth from your values or your Secret
Bearer token	`useEnrolmentToken: false` with `bearerToken.existingSecret` and `key`	`Authorization: Bearer`

Setting a bearer token together with any basic-auth credential is refused by the chart.

With the default mode, your write ingress must check basic auth against an htpasswd file. The plane can maintain that file for you from the cluster registry. The reactor writes one <cluster-name>:<bcrypt-hash> line per cluster into a Secret, sorted by name, and patches it only when it changed.

Create the Secret

Create an empty Secret with an empty users key in the namespace your ingress reads it from. If something else populates that Secret today (a secret-manager sync), replace that with the plain Secret. Two writers on one key will fight.

Grant the plane access

Apply a Role in that namespace that allows get and patch on that one Secret by resourceNames, and bind it to the service account the reactor runs as. Do not grant create.

Enable it on the plane

env:
- name: HANGAR_REMOTE_WRITE_AUTH_ENABLED
  value: "true"
- name: HANGAR_REMOTE_WRITE_AUTH_NAMESPACE
  value: monitoring
- name: HANGAR_REMOTE_WRITE_AUTH_SECRET
  value: prom-remote-write-auth

HANGAR_REMOTE_WRITE_AUTH_KEY is the key inside the Secret and defaults to users. The namespace is required when the flag is on. With no in-cluster service account, the pass logs once and does nothing.

Only clusters whose enrolment token was issued after the plane gained this feature have a stored verifier. A cluster registered earlier is absent from the file and its agent gets 401 on every send, with no signal in the console. Re-issue its enrolment token and reinstall with the new values.

Ingress bcrypt verification runs on every write. The plane uses a low cost factor (8) because the token carries enough entropy, but a public write endpoint is still a CPU target. Rate-limit it in front of the auth middleware.

GET /v1/system/config on the plane reports the write URL and the Secret’s coordinates, which is the first place to check when the file is not being maintained.

The cluster label is not authenticated

The cluster label is whatever the cluster’s agent writes. The receiver checks the connection’s credential and does not check the label against it. A cluster holding valid write credentials can write series under any cluster label, including another cluster’s.

What per-cluster credentials give you is revocation: you can cut one cluster off without touching the others, and a decommissioned cluster’s line leaves the file when its registry row does. They do not make the label trustworthy. Do not use the label for billing or per-tenant views, and do not share one store between parties who should not overwrite each other’s series.

Console integration

The console reads metrics through the plane’s /v1/metrics/* routes. The plane queries the store, not the browser, and every query is narrowed by the cluster’s cluster label so one cluster’s panels never include another’s. A cluster with no registered label has no metrics source, and the console says so instead of guessing.

Retention

Wingsmith keeps no metrics itself. Retention of fleet metrics is whatever you configure on the store, and retention of the component series is whatever your Prometheus keeps.

No series from any component carries request or response content. The gateway’s access log line (gateway: request) carries the request id, method, path without the query, status, outcome, timings, token counts, tenant and a 12-character key-hash prefix. It never carries a body, header value or model answer.