Architecture

The module operates in two mutually exclusive modes, selected by the dra.enabled parameter:

• DRA mode (dra.enabled: true) — the recommended mode, in Preview; enabled explicitly. GPUs are requested through the Kubernetes API using ResourceClaim resources.
• Device Plugin mode — the default mode, it works without extra configuration. GPUs are requested through the nvidia.com/gpu extended resource, and sharing is configured per NodeGroup.

Both modes deploy into namespace d8-nvidia-gpu.

DRA mode

Components

The module separates a vendor-agnostic core from per-vendor adapters. The core knows nothing about any specific GPU brand; everything that touches vendor hardware lives behind a contract implemented by a vendor adapter.

In DRA mode the module creates the following components:

The module works through discovery: gpu-node-agent runs everywhere and labels each node with the vendor it found, and every vendor adapter is scheduled by a nodeSelector on that label. A node with no supported GPU gets no adapter at all.

Vendor adapters

A vendor adapter is a DaemonSet with two working containers plus a metrics guard:

The adapter implements a Backend contract with three mandatory methods: the vendor name, a capabilities reader, and a CDI writer. Everything beyond that is an optional capability interface, so a vendor implements only what its hardware can actually do:

What a vendor implements is visible in the opaque device configs its adapter registers. NVIDIA registers GpuConfig, MigDeviceConfig and VfioDeviceConfig; MetaX registers GpuConfig and VfioDeviceConfig, so whole cards and passthrough work there but partitioning does not. A claim carrying a config kind no adapter registered is rejected by the admission webhook. For module capabilities by vendor, see Module capabilities by GPU vendor.

To add a vendor, write a container against that contract, add its PCI vendor ID to the gpu-supported-vendors ConfigMap, and ship an adapter DaemonSet. The CRD does not change, and neither does any core component.

DRA workloads receive their GPUs through CDI injection under the default container runtime. The nvidia RuntimeClass that the module ships is used only by the DCGM DaemonSets — normal DRA workloads must not set runtimeClassName: nvidia.

Conditions for enabling the mode

Three conditions decide what gets deployed. They do not gate one and the same set of templates:

To find out which condition is blocking deployment, check the module conditions and the alerts:

d8 k get module gpu -o jsonpath='{.status.conditions}' | jq

DRA feature gates readiness

The DRA feature gates are enabled by the Deckhouse platform, not by this module. Before deploying the DRA stack, the module verifies that the gates are genuinely enabled: it reads their real state from the kube-apiserver metrics rather than inferring it from version numbers.

Required feature gates:

Three of them arrive only in Kubernetes 1.34, which is why DRA mode needs that version even though the DRA core itself works on 1.33.

When dra.enabled: true, the module deploys the DRA workload stack — the controller, the node agent, the vendor adapters and everything that serves a ResourceClaim — only if all required DRA feature gates are enabled. Otherwise that part is not deployed, and the module reports the reason through the module condition FeatureGatesReady=False and the alert D8GpuDraFirstEnablementBlocked (or D8GpuDraFeatureGatesCheckError, when the gate state could not be read at all). Device Plugin mode is not affected by this check.

Node preparation is gated separately, and this distinction matters on a cluster where the gates are missing. The NodeGroupConfiguration scripts apply as soon as dra.enabled: true and the migration is complete, without waiting for the gates — a node still gets its containerd drop-in, its sysctl and its readiness labels, because those are reversed when the module is disabled. The two steps whose effect is not reversed — the IOMMU kernel-command-line rewrite with its reboot request, and the vfio-pci binding — are withheld while the gates are confirmed absent and the workload stack has never been deployed. So “the gates are missing” means the workload stack stays away and the irreversible half of node preparation stays away, not that nothing is rendered at all.

If a required gate is switched off later while DRA mode is already active — for example by a manual change or an incomplete upgrade — the alert D8GpuDraFeatureGateDisabledOnComponent appears. This alert also covers per-kubelet drift that an apiserver-only check cannot detect.

How the module works when starting a Pod

Starting a Pod that requests a GPU includes the following stages:

1. gpu-node-agent discovers PCI devices on the node and creates a PhysicalGPU for each card, adding the labels gpu.deckhouse.io/vendor, gpu.deckhouse.io/device, gpu.deckhouse.io/node, and the supported sharing strategies (gpu.deckhouse.io/sharing.mps, gpu.deckhouse.io/sharing.ts).
2. The vendor adapter for that card reads its capabilities through the vendor library — NVML on NVIDIA — fills in .status.capabilities of the PhysicalGPU object (architecture, compute API, driver version, supported partition profiles), and moves the object to Ready.
3. Once the PhysicalGPU becomes Ready, dra-plugin publishes ResourceSlice objects — one set of devices per node — describing the whole card, every possible partition, and their sharing variants.
4. An administrator optionally creates a GPUClass; gpu-controller turns it into a set of DeviceClass objects that users request by name or through an extended resource.
5. The user creates a ResourceClaim (usually via a ResourceClaimTemplate) and references it from a Pod. The scheduler matches the claim against published devices.
6. On the chosen node, dra-plugin calls NodePrepareResources and passes the request to the vendor adapter, which performs any reconfiguration required — on NVIDIA that means creating a MIG partition, starting an MPS control daemon, or binding the card to vfio-pci — then returns a CDI device name.
7. The container starts with the provided GPU. On Pod deletion the reverse path runs, and any partition created for the claim is destroyed.

Published device attributes

Devices in a ResourceSlice carry attributes in the gpu.deckhouse.io domain, and their names encode whether the device is a whole card, a MIG partition, or a sharing variant. See Device attributes and Getting information about the allocated GPU.

Every device declares bindsToNode: true with bindingConditions: ["Ready"] and bindingFailureConditions: ["BindingFailed"], so a failed preparation surfaces as a BindingFailed condition on the claim rather than a silently pending Pod.

Causes of MIG layout conflicts

MIG partitions are modelled as partitionable devices. Each node publishes a shared counter set per card (slices, slice-0…slice-7, multiprocessors, copy-engines, decoders, jpeg-engines, ofa-engines), and every device — the whole card and each possible partition — declares which counters it consumes.

Consequences:

• The scheduler enforces mutual exclusion itself. Allocating a 7g partition consumes all slice counters, so no other partition on that card can be allocated at the same time.
• MIG partitions are ephemeral. A partition exists only between NodePrepareResources and NodeUnprepareResources. There is no persistent MIG layout to manage: the layout is a consequence of what is currently claimed.

PhysicalGPU lifecycle

PhysicalGPU lifecycle phases appear in .status.phase:

The phase is derived, never set directly. Five conditions carry the detail:

Devices are published only while the phase is Ready, so an AdapterReady=False card never appears in any ResourceSlice.

Phase derivation reads the driver binding first: a card whose .status.driverBinding.mode is Passthrough becomes Passthrough regardless of its conditions, because a vfio-pci-bound card legitimately has no adapter readiness and would otherwise be misread as Initializing. For every other mode the condition table applies, so returning the card to a compute driver leaves the phase naturally.

Note that a passthrough request on a virtualised node fails rather than reaching Passthrough: it lands on AdapterReady=False/VFIORequiresBareMetal, which derives to Failed.

Reconfiguration progresses through Requested → Draining → InProgress → Completing → None. When a hardware transition needs the node itself disrupted, the vendor adapter raises DisruptionRequired and the controller carries it through DisruptionPending to RebootRequested.

Driver binding

.status.driverBinding records what the kernel actually has bound, and who owns the device. Its fields have split ownership through Server-Side Apply:

In kubectl get physicalgpus output, the last two fields appear as the Mode and Management columns.

Device management state

managementState answers whether the module owns a device, and it is the only value that gates reclaim. The Managed condition carries the extensible reason behind it:

Opt a device out with the gpu.deckhouse.io/unmanaged annotation. A stray binding must hold continuously for a two-minute grace period before it is classified Orphaned, which keeps a brief race from raising a false orphan.

Request a reset with the gpu.deckhouse.io/reclaim annotation. The controller reflects progress on the Reclaim condition and refuses the request — with a reason on the condition rather than silently — when an active ResourceClaim still references the device, when the device is not Orphaned, or when its state cannot be determined.

A reclaim that starts but does not finish ends at Failed: if the card is still on a passthrough driver two minutes after the reset began, the condition reports that driverBinding.mode is still Passthrough and that a node reboot may be required. That is the outcome to watch for — the module cannot resolve it on its own.

Admission webhooks

Served by gpu-controller:

Additionally, a ValidatingAdmissionPolicy named gpu-vfio-switch-guard constrains the privileged Jobs that nvidia-adapter creates for VFIO driver switching (see below). Any Job in d8-nvidia-gpu that is privileged or uses hostPID — either one is enough to trigger the check — must match the sanctioned profile in full:

• the vfio-switch ServiceAccount, with no token automount
• exactly one container, running the exact adapter image
• hostNetwork and hostIPC both false
• restartPolicy: Never
• host paths limited to /sys, /dev, /proc, and /var/lib/nvidia-adapter/checkpoint
• a command of the form /app/nvidia-adapter switch <pci-address>, with a canonical PCI address

VFIO passthrough

For passthrough, the card must be detached from the NVIDIA driver and bound to vfio-pci. nvidia-adapter does not do this in its own process: it launches a short-lived privileged Job (/app/nvidia-adapter switch <pci>) running under a ServiceAccount with no Kubernetes permissions, whose only purpose is to be an admission identity. The gpu-vfio-switch-guard policy above is the guard on that capability.

Node preparation is handled by NodeGroupConfiguration (see below). A card in passthrough reports .status.phase: Passthrough and is not published as a CUDA device.

When a driver switch fails

Rebinding a card between drivers is the one operation that can leave hardware in a state Kubernetes cannot repair on its own, so it has an explicit recovery path.

A failed bind to vfio-pci is normally rolled back to the NVIDIA driver and the claim simply fails. Rollback is impossible in the following cases:

• after a failed or wedged unbind the card is left with no driver at all;
• on release, the card cannot be restored to the NVIDIA driver and stays on vfio-pci or driverless.

In both cases the adapter publishes Reconfiguring=DisruptionRequired on the PhysicalGPU with a message naming the PCI address and stating that a node reboot is required. gpu-controller then carries the handoff through DisruptionPending to RebootRequested. Once recovery observes the card back on its baseline driver — ground truth that the reboot actually happened — it finalises the condition through Completing.

Failures also surface as events: VFIOPrepareFailed and VFIOUnprepareFailed, alongside VFIOPrepareSucceeded on the happy path.

Node preparation

NodeGroupConfiguration scripts run on every node, in weight order. They are what makes a node eligible for the GPU stack.

Node labels that gate scheduling

If DRA pods are missing from a node, one of these labels is usually the reason:

Uninstall

When the module is disabled or removed while dra.enabled: true, a Helm pre-delete hook Job deletes every PhysicalGPU in the cluster and waits up to 600 seconds for them to go away. This lets gpu-controller resolve its finalizers before the CRD is removed.

If the Job times out, look for objects with stuck finalizers:

d8 k get physicalgpus -A

Device Plugin mode

In this mode dra.enabled is false, which is the default. GPU sharing is configured per NodeGroup through spec.gpu rather than per workload.

Components

In Device Plugin mode the module creates the following components:

Key labels

• node.deckhouse.io/gpu="" — set by the module hook on nodes of GPU Node Groups.
• node.deckhouse.io/device-gpu.config=<Exclusive|TimeSlicing|MIG> — set by the hook; selects the device plugin and GFD configuration.
• node.deckhouse.io/gpu-setup-complete="" — set by gpu-sysctl.sh after local checks; until it appears, the stack is not scheduled onto the node.
• feature.node.kubernetes.io/pci-*.present=true — published by NFD.
• nvidia.com/* (for example nvidia.com/gpu.count, nvidia.com/mig.capable) — published by NFD from GFD feature files.
• nvidia.com/mig.config=<profile|all-disabled> — set by the hook: the desired MIG profile, or all-disabled to roll MIG back.
• nvidia.com/mig.config.state=<pending|rebooting|success|failed> — set by nvidia-mig-manager.
• taint mig-reconfigure=true:NoSchedule — held by nvidia-mig-manager for the duration of a reconfiguration.
• annotations update.node.deckhouse.io/disruption-approved, .../draining, .../drained — controlled drain during MIG changes.

How the module works in Device Plugin mode

How the module works in Device Plugin mode:

1. The cluster administrator sets or changes spec.gpu in the NodeGroup.
2. Helm deploys or removes components in namespace d8-nvidia-gpu: GFD and device plugin as per-NodeGroup DaemonSets, NFD master/gc on the control plane, NFD worker and DCGM on GPU nodes.
3. The module hook updates node labels — adding node.deckhouse.io/gpu and device-gpu.config, and for MIG also nvidia.com/mig.config; on GPU disable it removes them and sets nvidia.com/mig.config=all-disabled to trigger rollback.
4. NodeGroupConfiguration scripts run in weight order: gpu-check.sh → gpu-runtime.sh → gpu-sysctl.sh. On success the node gets node.deckhouse.io/gpu-setup-complete.
5. The stack converges: NFD publishes labels, GFD writes GPU feature files, the device plugin exposes resources, DCGM starts exporting metrics.

Configuring GPU allocation with the Exclusive strategy

To allocate GPUs without sharing (each GPU is allocated entirely to one Pod), do the following:

1. Set spec.gpu.sharing: Exclusive.
2. The module sets the label node.deckhouse.io/device-gpu.config=Exclusive and brings up GFD and the device plugin.
3. The node exposes nvidia.com/gpu; each Pod gets a full GPU.

To disable, remove spec.gpu from the NodeGroup. The module removes its labels, NodeGroupConfiguration rolls back the runtime and sysctl changes, and NFD GC eventually cleans up the nvidia.com/* labels.

Configuring GPU allocation with the TimeSlicing strategy

To allocate GPUs with time-slicing (several Pods can use one GPU concurrently, taking turns on its compute resources), do the following:

1. Set spec.gpu.sharing: TimeSlicing and optionally spec.gpu.timeSlicing.partitionCount (default 4).
2. The device plugin applies the time-slicing configuration.
3. The node still exposes nvidia.com/gpu, but with more virtual slots.

Switching to Exclusive or removing spec.gpu reverts it.

Configuring GPU allocation with the MIG strategy

To allocate GPUs by splitting them into multiple instances (NVIDIA MIG, when supported), do the following:

1. Set spec.gpu.sharing: MIG and spec.gpu.mig.partedConfig: <profile> (for example all-1g.5gb).
2. The module sets the labels node.deckhouse.io/device-gpu.config=MIG and nvidia.com/mig.config=<profile>.
3. Once NFD publishes nvidia.com/mig.capable=true, nvidia-mig-manager is scheduled onto the node and reacts to changes of the nvidia.com/mig.config label.
4. To reconfigure, it sets nvidia.com/mig.config.state=pending, pauses GPU clients via nvidia.com/gpu.deploy.*=paused-for-mig-change, applies the taint mig-reconfigure=true:NoSchedule, waits for drain approval, deletes GPU client pods on the node, applies the profile (rebooting if required), then sets success or failed, removes the taint, uncordons the node, and restores the gpu.deploy.* labels.
5. The cluster then exposes resources such as nvidia.com/mig-1g.5gb.

If the nvidia-mig-manager Pod is restarted mid-operation, its preStop hook waits for the active operation to finish, then uncordons and removes the taint and drain annotations on a best-effort basis.

Rolling MIG back. Switching away from MIG or removing spec.gpu sets the label nvidia.com/mig.config=all-disabled, which the manager applies the same way. Once rollback succeeds, the labels nvidia.com/mig.config and nvidia.com/mig.config.state are removed so the manager does not keep waiting on the node. The manager never runs on GPUs without MIG support (nvidia.com/mig.capable=false) — use Exclusive or TimeSlicing for those.

Module shared infrastructure

Regardless of operating mode, the module uses the following infrastructure:

Monitoring

The module exports its own metrics alongside the DCGM metrics:

Scrape configuration is created automatically: ServiceMonitors for gpu-controller and nvidia-adapter, and a PodMonitor for the DCGM exporter. It can be turned off with monitoring.serviceMonitor.

Grafana dashboards are shipped for both modes — cluster overview, per-node and per-namespace views, GPU inventory, and DRA workloads.

Alerts: