Devices and Topology

Device identity from DCGM, device classes, health, topology domains, and capacity.

A device is one accelerator, identified by its GPU UUID. Wingsmith schedules at the device level, not the node level, because “eight GPUs” can mean eight healthy cards in one NVLink domain or seven healthy plus one throttling across four chassis. Every capacity and placement question that matters is per device.

Device identity

Device identity comes from the DCGM exporter each cluster already runs, not from Kubernetes labels. The operator reads DCGM’s Prometheus metrics inside the cluster and reports each device to the control plane.

Each device carries:

• uuid: The GPU UUID, unique across the fleet
• cluster_id: Which cluster this device belongs to
• node_id: Which node it sits in
• pci_bus_id: PCI address for topology
• device_class: What kind of accelerator this is (e.g., h100-80gb-sxm)

Kubernetes node labels say how many GPUs a node has and what model, but not which GPUs. The label nvidia.com/gpu.count=8 doesn’t tell you which of the eight is throttling or allocated. DCGM gives you the per-device UUID, health, and memory.

Device classes

A device class is Wingsmith’s name for a kind of accelerator. The class groups devices by their scheduling-relevant properties: architecture, memory, form factor. Examples: h100-80gb-sxm, a100-80gb-pcie, l40s-48gb-pcie.

Device classes are defined in libs/packages/contract/devices.json, a shared table read by both the control plane and the operator. Vendor strings from DCGM (like NVIDIA H100 80GB HBM3) map to a canonical class through aliases in that file. Adding new hardware means updating devices.json and releasing both components so they agree on the mapping.

Each class has:

• class: The identifier (what travels on the wire)
• label: The human-readable name (what you show in a UI)
• architecture: Vendor generation (hopper, ampere, ada, blackwell)
• memory_mib: Nameplate memory in MiB
• form_factor: Physical package (sxm, pcie, nvl, server-edition)
• devices_per_node: How many devices share a coherent interconnect (default 8; GB200 NVL72 is 72)

Workloads declare a required device_class and optional device_memory_mib_min. Placement only considers clusters with free devices of that class meeting the memory floor.

Health states

Each device reports its health. The health field is one of:

• healthy: Normal operation, no errors
• unhealthy: ECC errors, thermal throttling, or other degraded state
• offline: Not reachable or not responding

The operator reads health from DCGM metrics like DCGM_FI_DEV_ECC_DBE_VOL_TOTAL (uncorrectable ECC errors) and DCGM_FI_DEV_THERMAL_VIOLATION (throttle state). A device with active throttling is marked unhealthy.

Placement counts only healthy devices as capacity. An unhealthy device doesn’t contribute to a cluster’s free count, even if it’s not allocated.

Memory and ECC errors

Each device reports:

• memory_total_mib: Total VRAM
• memory_free_mib: Currently unallocated
• ecc_errors: Count of uncorrectable ECC errors since boot

ECC errors are cumulative. A device with non-zero ecc_errors is marked unhealthy and should be cordoned or reset.

MIG profiles

NVIDIA MIG (Multi-Instance GPU) partitions one physical GPU into isolated slices. If a cluster runs MIG, the operator reports each MIG instance as a separate device with its own UUID and device_class.

For example, an H100 80GB partitioned as 1g.10gb slices appears as eight devices, each of class h100-mig-1g-10gb (or whatever alias devices.json maps the MIG profile to). Workloads can request MIG devices by class, and placement treats them as distinct from the full GPU.

MIG is a cluster-level capability. The operator reports capabilities.mig: true if the cluster supports it.

Topology domains

A topology domain groups devices by their interconnect. Each domain has a kind:

• nvlink: Devices connected via NVLink or NVSwitch fabric
• pcie: Devices on the same PCIe root complex
• numa: Devices in the same NUMA node

The operator reads topology from DCGM (DCGM_FI_DEV_NVLINK_* metrics) and reports it as TopologyDomain objects. Each domain lists the device UUIDs it contains.

Workloads requiring tensor parallelism declare requires_single_topology_domain: true and optionally a topology_domain_kind: nvlink. Placement then only considers clusters where the needed device count fits inside one domain of that kind. A workload that needs eight GPUs in one NVLink fabric is refused from a cluster where the eight GPUs are scattered across four chassis.

Modeling topology as objects from the start means “eight GPUs in one fabric” is a query over observed state, not an annotation that gets retrofitted later.

Allocation

An allocation is the observed binding of devices to a replica. When a pod starts and claims GPUs, the operator reports which devices it got.

Each allocation carries:

• replica_id: Which replica holds the devices
• device_ids[]: UUIDs of the allocated devices
• node_id: Which node the replica runs on

Allocations let you answer “which devices does this replica use” and “which replica is on this device.” The control plane reads them to count how many devices a project is using for quota enforcement.

A device is allocated when it appears in a live allocation. A device is allocatable (free and claimable) when it’s healthy, schedulable, and not allocated.

Capacity and device counts

The control plane counts devices to answer placement and quota questions. Two counts matter:

Free devices: healthy and allocatable (not held by any replica). This is what placement checks when admitting a new deployment.

Allocated devices: Held by a live replica. This is what counts against a project’s quota.

A node with eight devices where one is unhealthy and three are allocated has four free devices. Placement can admit a workload needing four or fewer devices of that class on that node.

Node-level GPU counts like nvidia.com/gpu.count=8 don’t express “seven of eight healthy” or “which GPU has ECC errors.” That’s why Wingsmith models devices as first-class objects.