Moving GPU workloads to DRA

Both GPU management stacks run in the cluster at the same time, split across nodes, so the migration is done gradually: node by node, application by application. Workloads that have not been moved yet keep running on their existing nodes unchanged.

How the stacks are split across nodes

A node is served by exactly one stack, and the stack is determined by the node’s group.

• Nodes of a NodeGroup that has the spec.gpu section are served by the Device Plugin stack.
• With dra.enabled: true, all other nodes are served by DRA. The exceptions are control plane nodes and nodes labeled gpu.deckhouse.io/enabled=false.

The two stacks cannot coexist on one node: the Device Plugin stack makes the NVIDIA runtime the default one, while DRA keeps the ordinary runtime and passes devices in through CDI. The unit of migration is therefore a node group, and the migration itself is moving a node from one group to another.

If DRA was enabled before the module upgrade

This section applies to clusters that ran with dra.enabled: true before the module upgrade and that have node groups with spec.gpu. In other clusters, groups with spec.gpu are served by the Device Plugin stack right away.

In earlier versions, with dra.enabled: true DRA served every GPU node, and the spec.gpu section had no effect. To keep the upgrade from moving these groups to the Device Plugin stack together with their running claims, the module records every group that has spec.gpu on the first run of the new version and keeps it on DRA. The D8GpuNodeGroupHeldOnDra alert fires for each such group.

A group is released when its spec.gpu is removed: it stays on DRA, and the alert clears. Disabling dra.enabled releases all groups. To move a held group to the Device Plugin stack, drain its nodes, remove spec.gpu, wait for the alert to clear, and set spec.gpu again.

Prerequisites

• Kubernetes version is 1.34 or newer.
• The feature gates are enabled: DynamicResourceAllocation, DRAResourceClaimDeviceStatus, DRADeviceBindingConditions, DRAConsumableCapacity, DRAPartitionableDevices, DRAExtendedResource — the platform enables them, and the module changes neither control plane nor kubelet arguments.
• The NVIDIA driver and the NVIDIA Container Toolkit are installed on the nodes.
• There is enough spare capacity to take at least one node out of service.

Gate readiness is visible in the module status:

d8 k get module gpu -o jsonpath='{.status.conditions}' | jq

Step 1. Create the target group and enable DRA

Create the group empty and without the spec.gpu section.

apiVersion: deckhouse.io/v1
kind: NodeGroup
metadata:
  name: gpu-dra
spec:
  nodeType: Static

Copy the remaining parameters from the source group: taints, labels, kubelet settings. A mismatch shows up later and not where it will be looked for — the application simply behaves differently on the new nodes.

Then enable DRA:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: gpu
spec:
  enabled: true
  version: 1
  settings:
    dra:
      enabled: true

DRA components are deployed on the nodes of groups without spec.gpu, including the new gpu-dra group. GPU nodes in other groups without spec.gpu, which no stack served before, get DRA right away as well. On nodes of groups with spec.gpu the Device Plugin stack keeps working.

Step 2. Create a GPU class

A class describes which cards are available to applications and on what terms.

apiVersion: gpu.deckhouse.io/v1alpha1
kind: GPUClass
metadata:
  name: a100
spec:
  selector:
    matchLabels:
      gpu.deckhouse.io/device: a100-sxm4-40gb

Take the gpu.deckhouse.io/device label value from the PhysicalGPU objects of your own cluster. Partition and sharing filters are described in GPU classes.

The controller derives allocation variants from the class — a whole card, a MIG partition, a share of a card — and publishes each of them as its own extended resource. These are the names applications request:

d8 k get gpuclass a100 -o jsonpath='{.status.deviceClassNames}' | jq

Step 3. Hand a node over

Drain the node first: its stack changes, and so does the way its cards are published.

1. d8 k cordon <node>
2. d8 k drain <node> --ignore-daemonsets --delete-emptydir-data
3. Move the node to the gpu-dra group.
4. Wait for node preparation and check readiness.
5. d8 k uncordon <node>

How the node is moved depends on how it joined the cluster. A manually added node is moved by changing its group label, with no reinstall:

d8 k label node --overwrite <node> node.deckhouse.io/group=gpu-dra
d8 k label node <node> node-role.kubernetes.io/<old-group>-

Nodes managed by a StaticInstance and cloud nodes cannot be moved this way — they are recreated by their own group.

After the label changes, the node is reconfigured by the next bashible run: its containerd configuration changes, so containerd is restarted. Running containers survive that restart, but GPU workloads are still drained in advance — the cards of the node are published by a different stack afterwards.

The node is ready when all four conditions hold.

• The node.deckhouse.io/gpu-setup-complete label is present.
• The node.deckhouse.io/gpu-stack=dra label is present.
• A PhysicalGPU object in the Ready phase exists for every card.
• The gpu-node-agent and vendor adapter Pods are running on the node.

d8 k get physicalgpu -o wide
d8 k -n d8-nvidia-gpu get pods -o wide

If the node has been moved to a group without spec.gpu while dra.enabled is off, no stack manages it: the cards of such a node sit idle.

Step 4. Move an application

Two things change in the manifest. The nvidia.com/gpu request is replaced with the extended resource of a class, and the binding to the source group is removed — the node is chosen by the requested class.

# before
      nodeSelector:
        node.deckhouse.io/group: gpu
          resources:
            limits:
              nvidia.com/gpu: 1

# after
          resources:
            limits:
              gpu.deckhouse.io/<class>: 1

ResourceClaim objects do not go into the manifest: the module creates them when the Pod is created. Everything else stays as it was. Other ways to request a device are covered in Usage and Manifest examples.

The change takes effect when a Pod is created, so restart the workload after editing the manifest:

d8 k -n <namespace> rollout restart deployment <name>

Step 5. Verify the result

d8 k -n <namespace> get pods -o wide
d8 k -n <namespace> get resourceclaims
d8 k -n <namespace> exec <pod> -- nvidia-smi -L

The application has been moved if its Pods are placed on nodes of the target group, the claims are allocated, and the card is visible inside the container. A Pod in Pending means there are no free devices of the requested kind in the target group — hand over the next node.

Steps 3 and 4 repeat while the source group still has nodes and workloads. The order inside an iteration matters: hand over the node before moving the application, otherwise its Pods wait for devices that do not exist in the cluster yet.

Completion

1. Hand over the remaining nodes and move the remaining applications.
2. Delete the source node group, or remove the spec.gpu section from it.
3. Replace the requests.nvidia.com/gpu limits in ResourceQuota with limits on DRA devices, keeping the same values.
4. Clear the leftover Device Plugin labels — node.deckhouse.io/device-gpu.config and the nvidia.com/* labels from NFD. DRA mode ignores them, but they make node state harder to read.

Rollback

Rollback happens at two independent levels.

An application is rolled back by editing its manifest — back to the nvidia.com/gpu request and the binding to the source group — and restarting the Pods. A node is rolled back by moving it into a group with spec.gpu, in the same order as the handover: cordon, drain, group change, verification.

A rollback needs Device Plugin nodes, so delete the source group last — only after the migration is deemed successful and the trial period on DRA is over.

Common mistakes