Metrics

Metrics reference

Every Prometheus series the Wingsmith plane, gateway, operator and agent export, with labels and meaning.

Four components serve Prometheus text format on their own listener. To scrape them and read the fleet’s metrics in the console, see Monitoring.

Component	Endpoint	Notes
Plane api	`:8000/metrics`	No credential. Each pod reports only its own requests, so scrape every pod
Gateway	`:9090/metrics`	`HANGAR_GATEWAY_METRICS_LISTEN`; `off` disables
Operator	`:8080/metrics`	Chart value `operator.metrics.enabled`, on by default
Agent	`:8081/metrics`	Shares `HANGAR_HEALTH_ADDR` with the probes

The plane’s hub and reactor processes export nothing yet. No series carries request or response content, caller identity, API keys or client addresses.

Plane

Series	Type	Labels	What
`hangar_plane_http_requests_total`	counter	`method`, `route`, `status`	Requests answered. `route` is the matched template, such as `/v1/clusters/{ref}`
`hangar_plane_http_request_duration_seconds`	histogram	`method`, `route`	Time from the request reaching the api to the end of its response. Default client buckets, 5 ms to 10 s
`hangar_plane_http_requests_in_flight`	gauge		Requests being answered now
`hangar_plane_build_info`	gauge	`version`	Always 1. `version` is the `hangar-plane` package version
`hangar_plane_db_pool_connections`	gauge	`state` (`checked_out`, `idle`)	Connections in the database pool. Absent until the process opens its pool
`hangar_plane_db_pool_size`	gauge		Configured pool size (`HANGAR_DB_POOL_SIZE`). Absent until the pool opens
`hangar_plane_db_pool_overflow`	gauge		Connections open beyond the pool size, up to `HANGAR_DB_POOL_MAX_OVERFLOW`. Negative while the pool is not full
`process_`, `python_info`, `python_gc_`	various		Process and interpreter series. `process_*` is Linux only

Requests that match no route (a 404 before routing) are not counted.

Useful queries:

# Error ratio by route over five minutes
sum by (route) (rate(hangar_plane_http_requests_total{status=~"5.."}[5m]))
  / sum by (route) (rate(hangar_plane_http_requests_total[5m]))

# p95 latency by route
histogram_quantile(0.95, sum by (route, le) (rate(hangar_plane_http_request_duration_seconds_bucket[5m])))

# Pool in use beyond its size
hangar_plane_db_pool_connections{state="checked_out"} >= on(job, instance) hangar_plane_db_pool_size

The gateway counts every request, including ones that never reach a cluster. Request series share the labels hangar_endpoint, endpoint_name and origin. The routed series add deployment and cluster, which are empty strings when a request never got that far.

Requests

Series	Type	Extra labels	What
`hangar_gateway_requests_total`	counter	routed, `status`, `outcome`	Requests answered
`hangar_gateway_request_duration_seconds`	histogram	routed	Time this hop spent on a request, first byte in to last byte out. Buckets 50 ms to 300 s
`hangar_gateway_ttft_seconds`	histogram	routed	Arrival to first response byte forwarded. Only requests that reached a cluster
`hangar_gateway_ttft_by_input_seconds`	histogram	common, `input_bucket`	Time to first byte banded by prompt length. `unknown` when the request never disclosed one
`hangar_gateway_requests_in_flight`	gauge	routed	Requests being proxied to a cluster now
`hangar_gateway_upstream_replays_total`	counter	routed	Request bodies sent to a cluster again because the transport retried
`hangar_gateway_upstream_up`	gauge	routed	1 when the plane’s last table reported the deployment running. This is the plane’s observation, not a probe
`hangar_gateway_tokens_total`	counter	routed, `direction`	Tokens from the upstream’s own usage object. `direction` is `input`, `output` or `cached_input`
`hangar_gateway_generated_total`	counter	routed, `unit`	Image and video output counted by the gateway: `images`, `videos`, `video_seconds`, `video_frames`, `upstream_seconds`

Token and generation counts are observability, not metering. Do not reconcile them against an invoice.

origin is live for callers and probe for the conformance suite. outcome is one of ten values: ok, client_error, auth_rejected, rate_limited, shed, no_upstream, upstream_error, upstream_timeout, canceled, internal_error. rate_limited and shed can both be HTTP 429 (shed can also be a 503), but only shed is a failure, so key failure rates off outcome, not status.

hangar_endpoint is _catalog for GET /v1/models and _unknown for a model the gateway does not serve.

Routing table

Series	Type	Labels	What
`hangar_gateway_routing_table_age_seconds`	gauge		Seconds since the plane last confirmed the loaded table. Grows while the plane is unreachable
`hangar_gateway_routing_table_confirmed_timestamp_seconds`	gauge		Unix time of that confirmation
`hangar_gateway_routing_table_refresh_success`	gauge		1 if the last refresh reached the plane, 0 if not
`hangar_gateway_routing_table_refreshes_total`	counter	`result` (`applied`, `unchanged`, `error`)	Refresh attempts. A failed refresh keeps the loaded table
`hangar_gateway_routing_table_endpoints`	gauge		Endpoints in the loaded table

Tunnels

Series	Type	Labels	What
`hangar_gateway_tunnels`	gauge	`cluster_id`	Live tunnels this replica holds to a cluster
`hangar_gateway_tunnel_coverage`	gauge	`cluster_id`, `cluster`	1 if this replica can reach the cluster, 0 if its routing table routes there but it cannot
`hangar_gateway_tunnel_streams_active`	gauge	`cluster_id`	Requests open on this replica’s tunnels to a cluster
`hangar_gateway_tunnel_streams_capacity`	gauge	`cluster_id`	Stream limits the agents advertised, summed. When active meets capacity, new requests get `503 no_capacity`
`hangar_gateway_tunnel_dials_rejected_total`	counter	`reason`	Dials that did not become tunnels. Reasons include `unauthorized`, `unavailable` (plane unreachable), `limit` and `throttled`
`hangar_gateway_tunnel_dials_duplicate_total`	counter		Dials that landed on this replica after the agent already covered it
`hangar_gateway_tunnels_revoked_total`	counter	`cluster_id`	Open tunnels closed because the plane no longer accepts their token
`hangar_gateway_tunnel_retries_total`	counter	`cluster_id`, `outcome`	Requests whose tunnel died under them: resent on another tunnel, `written` (not resent, the agent may have forwarded it) or `exhausted`
`hangar_gateway_tunnel_certificate_expiry_timestamp_seconds`	gauge		Unix time the tunnel certificate loaded in memory expires. Alert as it nears now
`hangar_gateway_build_info`	gauge	`version`	Always 1

The gateway also exports the standard Go runtime and process series.

Operator

The operator registers no series of its own yet. It exports controller-runtime’s standard families. A read-only operator runs no reconciler, so the controller_runtime_* families are absent there.

Series	Labels	What
`controller_runtime_reconcile_total`	`controller`, `result`	Reconciles by outcome
`controller_runtime_reconcile_errors_total`	`controller`	Reconciles that returned an error
`controller_runtime_terminal_reconcile_errors_total`	`controller`	Errors that will not be retried
`controller_runtime_reconcile_panics_total`	`controller`	Reconciles that panicked
`controller_runtime_reconcile_time_seconds`	`controller`	Reconcile latency (histogram)
`controller_runtime_max_concurrent_reconciles`, `controller_runtime_active_workers`	`controller`	Worker capacity and use
`workqueue_depth`, `workqueue_adds_total`, `workqueue_retries_total`	`name`, `controller`	Queue size and churn
`workqueue_queue_duration_seconds`, `workqueue_work_duration_seconds`	`name`, `controller`	Time waiting and time working (histograms)
`workqueue_unfinished_work_seconds`, `workqueue_longest_running_processor_seconds`	`name`, `controller`	Stuck work
`rest_client_requests_total`	`code`, `method`, `host`	Kubernetes API calls
`leader_election_master_status`	`name`	1 on the replica holding the lease
`go_`, `process_`		Go runtime and process

Hub-session health is not a metric. It is the operator’s /readyz, and the plane reports it per cluster.

Agent

Series	Type	Labels	What
`hangar_agent_requests_total`	counter	`outcome`, `code`	Requests that arrived over a tunnel. `forwarded` with a 5xx code is the frontend’s own error, `frontend_unavailable` is a frontend that did not answer, `aborted` is a response cut after it started
`hangar_agent_requests_in_flight`	gauge		Requests being served across all tunnels
`hangar_agent_request_duration_seconds`	histogram		Tunnel arrival to last response byte. Same buckets as the gateway
`hangar_agent_gateways_expected`	gauge		Gateway replicas the gateway last reported it was running
`hangar_agent_tunnels`	gauge	`gateway`	Established tunnels to each gateway replica
`hangar_agent_gateway_up`	gauge	`gateway`	1 when at least one tunnel to the replica is established
`hangar_agent_dial_attempts_total`	counter		Outbound dials attempted
`hangar_agent_dial_failures_total`	counter	`reason` (`dial`, `unauthorized`, `handshake`, `refused`)	Dials that did not produce a tunnel
`hangar_agent_dial_duplicates_total`	counter	`gateway`	Dials answered by a replica that already had its quota of tunnels. Not a failure
`hangar_agent_tunnels_established_total`	counter	`gateway`	Tunnels that completed the preface
`hangar_agent_tunnels_closed_total`	counter	`gateway`	Tunnels that ended
`hangar_agent_tunnel_seconds`	histogram	`gateway`	Tunnel lifetime. Buckets 1 s to 12 h
`hangar_agent_frontend_discovery_age_seconds`	gauge		Seconds since frontend discovery last got an answer from the API server

Coverage is complete when the number of hangar_agent_gateway_up == 1 series equals hangar_agent_gateways_expected.