FAQ

How do I delete a workload that has a GPU allocated?

To delete a workload that uses a GPU, run:

d8 k delete job gpu-hello

This deletes the related Pods and frees the GPU.

How do I install the NVIDIA driver on GPU nodes?

The NVIDIA driver and the NVIDIA Container Toolkit are installed through a NodeGroupConfiguration. Set nodeGroups to match your GPU NodeGroup.

The list of platforms supported by the NVIDIA Container Toolkit is in the NVIDIA documentation.

Installing the driver on Ubuntu

Tested on Ubuntu 24.04.

apiVersion: deckhouse.io/v1alpha1
kind: NodeGroupConfiguration
metadata:
  name: install-cuda-ubuntu.sh
spec:
  bundles:
  - ubuntu-lts
  content: |
    #!/bin/bash
    set -e

    if ! command -v curl &> /dev/null || ! command -v wget &> /dev/null; then
      sudo apt update
      sudo apt install -y curl wget
    fi

    CUDA_KEYRING_DEB="cuda-keyring_1.1-1_all.deb"
    NVIDIA_GPG_KEY="/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg"

    sudo apt update

    if [ ! -f "$CUDA_KEYRING_DEB" ]; then
      wget -q https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/$CUDA_KEYRING_DEB
      sudo dpkg -i $CUDA_KEYRING_DEB
      sudo apt update
    fi

    if [ ! -f "$NVIDIA_GPG_KEY" ]; then
      curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
        sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
      curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
        sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
        sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
      sudo apt update
    fi

    if ! dpkg-query -W -f='${Status}' "linux-headers-$(uname -r)" 2>/dev/null | grep -q "ok installed"; then
      sudo apt install -y "linux-headers-$(uname -r)"
    fi

    if ! dpkg-query -W -f='${Status}' cuda-drivers 2>/dev/null | grep -q "ok installed"; then
      sudo apt install -y cuda-drivers
    fi

    if ! dpkg-query -W -f='${Status}' nvidia-container-toolkit 2>/dev/null | grep -q "ok installed"; then
      sudo apt install -y nvidia-container-toolkit
    fi

    if ! grep -q "nouveau.modeset=0" /etc/default/grub; then
      sudo sed -i 's/GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="nouveau.modeset=0"/' /etc/default/grub
      sudo update-grub
    fi

  nodeGroups:
  - gpu
  weight: 5

Installing the driver on Debian

Tested on Debian 13.

apiVersion: deckhouse.io/v1alpha1
kind: NodeGroupConfiguration
metadata:
  name: install-cuda.sh
spec:
  bundles:
  - debian
  content: |
    #!/bin/bash
    set -e

    export DEBIAN_FRONTEND=noninteractive

    # Preseed debconf answers to avoid interactive prompts.
    echo "keyboard-configuration keyboard-configuration/layout select English (US)" | debconf-set-selections
    echo "keyboard-configuration keyboard-configuration/model select Generic 105-key (Intl) PC" | debconf-set-selections
    echo "keyboard-configuration keyboard-configuration/variant select English (US)" | debconf-set-selections
    echo "console-setup console-setup/charmap select UTF-8" | debconf-set-selections

    sudo -E apt update

    if ! command -v curl &> /dev/null || ! command -v wget &> /dev/null; then
      sudo apt update
      sudo apt install -y curl wget
    fi

    CUDA_KEYRING_DEB="cuda-keyring_1.1-1_all.deb"
    NVIDIA_GPG_KEY="/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg"

    sudo apt update

    if [ ! -f "$CUDA_KEYRING_DEB" ]; then
      wget -q https://developer.download.nvidia.com/compute/cuda/repos/debian13/x86_64/$CUDA_KEYRING_DEB
      sudo dpkg -i $CUDA_KEYRING_DEB
      sudo apt update
    fi

    if [ ! -f "$NVIDIA_GPG_KEY" ]; then
      curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
        sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
      curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
        sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
        sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
      sudo apt update
    fi

    if ! dpkg-query -W -f='${Status}' "linux-headers-$(uname -r)" 2>/dev/null | grep -q "ok installed"; then
      sudo apt install -y "linux-headers-$(uname -r)"
    fi

    if ! dpkg-query -W -f='${Status}' cuda-drivers 2>/dev/null | grep -q "ok installed"; then
      sudo apt install -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" cuda-drivers
    fi

    if ! dpkg-query -W -f='${Status}' nvidia-container-toolkit 2>/dev/null | grep -q "ok installed"; then
      sudo apt install -y nvidia-container-toolkit
    fi

    if ! grep -q "nouveau.modeset=0" /etc/default/grub; then
      sudo sed -i 's/GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="nouveau.modeset=0"/' /etc/default/grub
      sudo update-grub
    fi

  nodeGroups:
  - gpu
  weight: 5

Installing the driver on CentOS

Tested on CentOS 9.

apiVersion: deckhouse.io/v1alpha1
kind: NodeGroupConfiguration
metadata:
  name: install-cuda.sh
spec:
  bundles:
  - centos
  content: |
    #!/bin/bash
    set -e
    INSTALL_NEEDED=false

    # Install drivers (first run only).
    if ! rpm -q nvidia-driver-cuda &> /dev/null; then
      dnf install -y curl epel-release dkms gcc make dracut kernel-devel-$(uname -r) elfutils-libelf-devel
      dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
      rpm --import https://developer.download.nvidia.com/compute/cuda/repos/GPGKEY
      dnf install -y nvidia-driver-cuda nvidia-driver-cuda-libs nvidia-settings nvidia-persistenced
      curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | tee /etc/yum.repos.d/nvidia-container-toolkit.repo
      dnf install -y nvidia-container-toolkit
      INSTALL_NEEDED=true
    fi

    # Force-rebuild the module for the current kernel and update initramfs.
    dkms autoinstall
    dracut --force

    # Configure module autoload and disable nouveau.
    printf '%s\n' 'nvidia' 'nvidia_uvm' 'nvidia_drm' > /etc/modules-load.d/nvidia.conf
    grubby --update-kernel ALL --args="nouveau.modeset=0"

    # Reboot if changes were made.
    if [ "$INSTALL_NEEDED" = true ]; then
      base64_timer="W1VuaXRdCkRlc2NyaXB0aW9uPWJhc2hpYmxlIHRpbWVyCgpbVGltZXJdCk9uQm9vdFNlYz0xbWluCk9uVW5pdEFjdGl2ZVNlYz0xbWluCgpbSW5zdGFsbF0KV2FudGVkQnk9bXVsdGktdXNlci50YXJnZXQK"
      echo "$base64_timer" | base64 -d | tee /etc/systemd/system/bashible.timer
      systemctl enable bashible.timer
      base64_bashible="W1VuaXRdCkRlc2NyaXB0aW9uPUJhc2hpYmxlIHNlcnZpY2UKCltTZXJ2aWNlXQpFbnZpcm9ubWVudEZpbGU9L2V0Yy9lbnZpcm9ubWVudApFeGVjU3RhcnQ9L2Jpbi9iYXNoIC0tbm9wcm9maWxlIC0tbm9yYyAtYyAiL3Zhci9saWIvYmFzaGlibGUvYmFzaGlibGUuc2ggLS1tYXgtcmV0cmllcyAxMCIKUnVudGltZU1heFNlYz0zaAo="
      echo "$base64_bashible" | base64 -d | tee /etc/systemd/system/bashible.service
      systemctl enable bashible.service
      systemctl reboot
    fi

  nodeGroups:
  - gpu
  weight: 5

Installing the driver on RED OS

Tested on RED OS 8.

apiVersion: deckhouse.io/v1alpha1
kind: NodeGroupConfiguration
metadata:
  name: install-cuda.sh
spec:
  bundles:
  - redos
  content: |
    #!/bin/bash
    set -e
    INSTALL_NEEDED=false

    # Install the NVIDIA drivers.
    if ! rpm -q nvidia-drivers &> /dev/null || ! rpm -q cuda &> /dev/null || ! rpm -q nvidia-persistenced &> /dev/null; then
      echo "Update system"
      sudo dnf update -y
      sudo dnf install -y nvidia-drivers.x86_64 cuda.x86_64 nvidia-persistenced.x86_64
      INSTALL_NEEDED=true
    fi

    # Install the NVIDIA Container Toolkit.
    if ! rpm -q nvidia-container-toolkit &> /dev/null; then
      echo "NVIDIA container toolkit is not installed. Installing..."
      curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
      sudo dnf install -y nvidia-container-toolkit
      INSTALL_NEEDED=true
    fi

    # Configure GRUB and disable the nouveau driver.
    if ! grep -q "nouveau.modeset=0" /etc/default/grub; then
      echo "GRUB configuration and nouveau disabling"
      sudo -E grubby --update-kernel ALL --args="nouveau.modeset=0"
      sudo grub2-mkconfig -o /boot/grub2/grub.cfg
      INSTALL_NEEDED=true
    fi

    # Create the bashible service if drivers were installed.
    if [ "$INSTALL_NEEDED" = true ]; then
      base64_timer="W1VuaXRdCkRlc2NyaXB0aW9uPWJhc2hpYmxlIHRpbWVyCgpbVGltZXJdCk9uQm9vdFNlYz0xbWluCk9uVW5pdEFjdGl2ZVNlYz0xbWluCgpbSW5zdGFsbF0KV2FudGVkQnk9bXVsdGktdXNlci50YXJnZXQK"
      echo "$base64_timer" | base64 -d | sudo tee /etc/systemd/system/bashible.timer
      sudo systemctl enable bashible.timer
      base64_bashible="W1VuaXRdCkRlc2NyaXB0aW9uPUJhc2hpYmxlIHNlcnZpY2UKCltTZXJ2aWNlXQpFbnZpcm9ubWVudEZpbGU9L2V0Yy9lbnZpcm9ubWVudApFeGVjU3RhcnQ9L2Jpbi9iYXNoIC0tbm9wcm9maWxlIC0tbm9yYyAtYyAiL3Zhci9saWIvYmFzaGlibGUvYmFzaGlibGUuc2ggLS1tYXgtcmV0cmllcyAxMCIKUnVudGltZU1heFNlYz0zaAo="
      echo "$base64_bashible" | base64 -d | sudo tee /etc/systemd/system/bashible.service
      sudo systemctl enable bashible.service
      sudo systemctl reboot
    fi

  nodeGroups:
  - gpu
  weight: 5

Installing the driver on Astra Linux

Tested on Astra Linux 1.8.

apiVersion: deckhouse.io/v1alpha1
kind: NodeGroupConfiguration
metadata:
  name: install-cuda-astra.sh
spec:
  bundles:
  - astra
  content: |
    #!/bin/bash
    set -e
    export DEBIAN_FRONTEND=noninteractive

    # Clean up.
    rm -f /etc/apt/sources.list.d/cuda* 2>/dev/null || true
    rm -f /etc/apt/sources.list.d/nvidia* 2>/dev/null || true

    apt update
    apt install -y curl wget linux-headers-$(uname -r) build-essential dkms

    # CUDA repository (Debian 12).
    wget -q https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-keyring_1.1-1_all.deb
    dpkg -i cuda-keyring_1.1-1_all.deb
    rm -f cuda-keyring_1.1-1_all.deb
    apt update

    # Container Toolkit repository.
    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
    curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
      sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' > /etc/apt/sources.list.d/nvidia-container-toolkit.list
    apt update

    # Install driver 580.
    apt install -y nvidia-driver-580 nvidia-settings nvidia-compute-utils-580 nvidia-container-toolkit

    # Disable nouveau.
    sed -i 's/GRUB_CMDLINE_LINUX=""/GRUB_CMDLINE_LINUX="nouveau.modeset=0"/' /etc/default/grub
    update-grub

  nodeGroups:
  - gpu
  weight: 5

After applying the configuration, reboot the nodes, then check the driver on the node with nvidia-smi.

How do I verify a GPU node?

The module adds labels to every prepared node.

To list the labels, run:

d8 k get node <node-name> -o json | jq -r '.metadata.labels | with_entries(select(.key | test("gpu")))'

A working GPU node must have all of the following labels:

To see the stack running on the node, run:

d8 k -n d8-nvidia-gpu get pods -o wide

Example output:

NAME                             READY   STATUS    RESTARTS   AGE
gpu-controller-7d9f8b6c4-xk2lp   2/2     Running   0          5m
gpu-dcgm-2gflv                   1/1     Running   0          5m
gpu-dcgm-exporter-vnjrq          1/1     Running   0          5m
gpu-node-agent-54gk8             1/1     Running   0          5m
nvidia-adapter-8qgqx             3/3     Running   0          5m

To see discovered hardware, run:

d8 k get physicalgpus -o wide

Every expected card must be Ready with HEALTHY=True. HEALTHY is a wide column, so it appears only with -o wide; the default output ends at AGE and does not carry HEALTHY.

To see devices offered to the scheduler, run:

d8 k get resourceslices

If PhysicalGPU objects exist but no ResourceSlice does, the cards are not in phase Ready: devices are published only in that phase. See What to do if PhysicalGPU objects exist but no ResourceSlice does.

How do I publish GPUs for users?

To publish GPUs for users, define a GPUClass. Users can then request a GPU by name:

apiVersion: gpu.deckhouse.io/v1alpha1
kind: GPUClass
metadata:
  name: a100
spec:
  selector:
    matchLabels:
      gpu.deckhouse.io/device: a100-sxm4-40gb

To list all DeviceClasses that the GPU controller created automatically from the GPUClass, run:

d8 k get gpuclass a100 -o jsonpath='{.status.deviceClassNames}' | jq

Share those names with users. Partition and sharing filters are described in GPU classes.

How do I migrate from Device Plugin mode to DRA?

To migrate, change the dra.enabled parameter from false to true. Migration runs automatically:

1. The check_migration hook detects the existing d8-nvidia-gpu namespace and adds the label gpu.deckhouse.io/managed-by=gpu to it.
2. On the next reconciliation migrationReady becomes true, and the DRA templates are rendered.
3. Helm removes the Device Plugin stack — device plugin, GFD, MIG manager, DCGM — and deploys the DRA stack in the same namespace.

After migration:

• spec.gpu on the NodeGroup no longer takes effect: nothing manages it anymore. Sharing is now chosen at the workload level. You can remove this field.
• Node labels from Device Plugin mode, such as node.deckhouse.io/device-gpu.config and the nvidia.com/* labels published by NFD, may linger for a while after NFD is removed. In DRA mode they are ignored.
• Workloads that request nvidia.com/gpu will stop scheduling because that resource is no longer published. Move them to claims before you switch, or plan for downtime.

Why did the DRA stack not deploy at all?

If after setting dra.enabled: true the d8-nvidia-gpu namespace has no gpu-controller, gpu-node-agent, or nvidia-adapter, one of the conditions required to render the DRA templates may not hold. All of the following must be true at once: dra.enabled, migrationReady, and draFeatureGatesReady.

Check the module status:

d8 k get module gpu -o jsonpath='{.status.conditions}' | jq

The usual cause is missing DRA feature gates. That shows up as FeatureGatesReady=False with reason FeatureGatesNotEnabled and the alert D8GpuDraFirstEnablementBlocked. The gates are enabled by the platform, not by this module, so the fix is to upgrade DKP. See DRA feature gates readiness.

What happens if a feature gate is disabled while DRA is running?

The alert D8GpuDraFeatureGateDisabledOnComponent fires when a required gate is switched off on the apiserver, the scheduler, or an individual kubelet. The kubelet case is the important one: the readiness check reads the apiserver only, so a kubelet that lost the gate is caught by this alert rather than by the readiness check.

What to do if a GPU node has no DRA pods?

Check the node labels:

d8 k get node <node-name> -o json | jq -r '.metadata.labels | with_entries(select(.key | test("gpu")))'

What to do if PhysicalGPU objects exist but no ResourceSlice does?

Devices are published only while a card is in phase Ready. Check the phase and conditions.

To list all physical GPUs in the cluster, run:

d8 k get physicalgpus

To inspect conditions for a specific GPU, run:

d8 k get physicalgpu <name> -o jsonpath='{.status.conditions}' | jq

AdapterReady=False with reason DriverNotReady and the message NVML capabilities not available for this device means the adapter could not talk to the card through NVML. Check the driver on the node and the nvidia-adapter container logs:

d8 k -n d8-nvidia-gpu logs daemonset/nvidia-adapter -c nvidia-adapter

A card in phase Passthrough is bound to vfio-pci and is intentionally not offered as a CUDA device.

What to do if a Pod stays Pending and the claim is never allocated?

No published device satisfies the claim. List all devices from ResourceSlice resources with their names and attributes:

d8 k get resourceslices -o json | jq '.items[].spec.devices[]? | {name, attributes}'

Common causes:

• The CEL selector matches nothing. A frequent mistake is asking for sharePercent while the selector excludes sharing variants with !has(...sharingStrategy) — those two conditions contradict each other. See Usage.
• An unguarded attribute in CEL. Referencing an optional attribute without has(...) makes the whole expression fail rather than evaluate to false.
• The requested MIG profile conflicts with a layout that another workload is currently holding. Partitions exist only while claimed, so the Pod waits until the holder finishes. This is the scheduler refusing to double-allocate the card’s slices, not an error.

What to do if GPU workloads fail although the driver and toolkit are installed?

Installed packages do not guarantee a working stack. In DRA mode (dra.enabled: true) the module does not just look for the components of the GPU host stack, it checks that the stack can actually serve containers: the kernel module is loaded, nvidia-smi answers, the kernel module and the driver are the same version, the container toolkit is present, nvidia-container-cli really sees the GPU, and the toolkit versions agree.

The check runs as an early bashible step on the node and never blocks node convergence: the node keeps receiving updates, and the step neither fails nor hangs (every call to a vendor binary is time-bounded, so a wedged kernel module cannot stall the run). What changes is:

• the readiness label node.deckhouse.io/gpu-setup-complete is removed, so the GPU stack components do not roll onto the broken node. The removal alone would not be enough — a later node-preparation step of the same run applies that label from its own state — so the verdict also raises a node-local bashbooster flag gpu-host-stack-unhealthy, and that flag is what makes the later step (gpu-setup.sh in DRA mode, gpu-sysctl.sh in Device Plugin mode) leave the label off. Nothing latches: both the label and the flag are recomputed from scratch by every node-preparation run, so no manual cleanup is needed after a fix;

• the node gets two labels with the verdict:

  d8 k get nodes -L node.deckhouse.io/gpu-host-stack-error,node.deckhouse.io/gpu-host-stack-vendor

• the module exports the gpu_node_host_stack_unhealthy{node,vendor,reason} metric and fires the D8GpuNodeHostStackUnhealthy alert after 15 minutes;

• the gpu_node_host_stack_unhealthy_nodes metric reports how many nodes are unhealthy and is always published, including with the value 0: it shows that the check is running even when nothing is broken.

The driver and the container toolkit are external prerequisites: the module diagnoses them, but never installs, upgrades or repairs them.

When does the check run, and how do I force a re-check?

This is a node preparation check, not a monitoring probe. bashible skips all steps when the node configuration checksum is unchanged, there is no reboot annotation and the node uptime has grown, so the verdict is recomputed on node bootstrap, on a configuration change (module settings, NodeGroup, a Deckhouse update) and after a reboot — not on a timer.

Two consequences:

• a stack that breaks after a successful converge — a driver/toolkit version drift introduced by a package update without a reboot, for example — is not noticed until the next preparation run;
• a repair does not clear the verdict by itself; the check has to run again.

To force a re-check on a node, either reboot it, or remove the checksum file and wait for the next bashible tick:

rm -f /var/lib/bashible/configuration_checksum

This limitation applies to the host software stack only. The state of the GPU devices themselves is tracked continuously and independently of node preparation: it is published in PhysicalGPU (status.driverBinding.mode for the observed driver, status.driverBinding.managementState for who manages the device), and a card left on a stray passthrough binding — no owning claim, no opt-out — is classified Orphaned once it has held that way for the two-minute grace period. The alert D8GPUPhysicalGPUOrphaned fires later still: it carries for: 15m, so it is raised about seventeen minutes after the binding went stray, not two.

In what order should I check the node?

Run this on the node itself, in this order — the checks go from the lowest layer of the stack upwards, and the first failing one is the root cause:

cat /proc/driver/nvidia/version                  # is the kernel module loaded?
nvidia-smi -L                                    # does the driver answer?
nvidia-smi --query-gpu=driver_version --format=csv,noheader
nvidia-container-cli info                        # does the container path work?
nvidia-ctk --version
nvidia-container-cli --version                   # do the toolkit versions match?

Details that the reason code does not carry are in the step log on the node. Where that log lives depends on the Deckhouse version, and this module supports both:

# Deckhouse 1.77 and newer — one file per bashible step:
cat /var/log/d8/bashible/step.006_gpu-check.sh.log

On Deckhouse 1.75 and 1.76 per-step logs do not exist. The step wrote to /var/lib/bashible/step.log, which every later step overwrites, and gpu-check runs at weight 6 — so by the end of a run its output is gone. Re-run the probes above by hand instead.

What do the NVIDIA reason codes mean?

Reason codes are vendor-specific: the pair (gpu-host-stack-vendor, gpu-host-stack-error) identifies the failure. GPUs of vendors for which the module has no checks yet are not probed and produce no labels.

The check makes no statement at all about a node that is outside the GPU scope: one that does not belong to a GPU NodeGroup and merely happens to carry a card (a control-plane node with a spare GPU, a virtualization host holding GPUs for passthrough), one where the GPUs are handed over to vfio-pci for passthrough, or one where there is no GPU of a supported vendor. Such a node gets no reason code and no alert, and any verdict left from a previous run is dropped silently, so a node does not keep alerting after its card has been pulled. A node that does belong to a GPU NodeGroup and has no vendor host stack is a different case and is reported as host-stack-not-installed.

After the fix, the next node-preparation run removes both gpu-host-stack-* labels, drops the gpu-host-stack-unhealthy flag, restores the readiness label, and the alert stops firing — see above on how to trigger that run if the fix required neither a reboot nor a configuration change.

What to do on RED OS 7.3 and other distributions with mixed package versions?

On RED OS MUROM 7.3 the updates repository ships several versions of the NVIDIA container packages at once (for example nvidia-container-toolkit 1.14.4 next to nvidia-container and nvidia-container-tools 1.18.0~rc). A plain installation happily mixes them, nvidia-smi on the host works, the card is visible — and GPU still does not work in containers. That is exactly the toolkit-version-mismatch case.

What to check:

rpm -qa 'nvidia-container*'
nvidia-ctk --version
nvidia-container-cli --version

All nvidia-container* packages must be of one version; pin the version explicitly during installation instead of relying on the repository default. Install the driver following the NVIDIA driver installation guide and the container toolkit following the NVIDIA Container Toolkit installation guide.

What to do if the container runs but has no GPU?

The most common bug in DRA manifests: spec.resourceClaims is declared, but the container has no matching resources.claims entry. The GPU is allocated and simply not handed to the container — with no error, no event, and no warning.

spec:
  resourceClaims:
    - name: gpu
      resourceClaimTemplateName: my-claim
  containers:
    - name: app
      resources:
        claims:
          - name: gpu     # Make sure this part is present.

What to do if nvidia-smi finds nothing inside a Pod?

If the Pod has no claim, this is correct behavior. DRA workloads run under the default container runtime and get devices only through a claim, so a Pod without one sees no /dev/nvidia* at all. Setting NVIDIA_VISIBLE_DEVICES=all does not change this — the variable is inert in this module.

If the Pod does have a claim, check the resources.claims entry, then confirm which device was allocated:

d8 k -n <namespace> get resourceclaim -o json \
  | jq '.items[].status.allocation.devices.results[] | {device, pool}'

What to do if an extended-resource request is ignored?

A Pod requests gpu.deckhouse.io/<class> and stays Pending, with the resource still visible in the Pod spec.

The mutating webhook that turns extended resources into claims deliberately skips kube-system and every d8-* namespace. In those namespaces the request is left untouched, and since no node advertises it, the Pod never schedules. Run GPU workloads in your own namespaces.

If the class name does not exist, you get an explicit rejection:

Error from server (Forbidden): admission webhook "podgpuresourceclaim.gpu.deckhouse.io" denied
the request: get DeviceClass "no-such-class": deviceclasses.resource.k8s.io "no-such-class" not found

Check the available names with d8 k get gpuclass <name> -o jsonpath='{.status.deviceClassNames}'.

What to do if creating or updating a GPUClass is rejected?

The GPUClass webhook has failurePolicy: Fail, so while gpu-controller is unavailable, writes are rejected. Check that it is running:

d8 k -n d8-nvidia-gpu get deploy gpu-controller

A validation error you may hit:

partitionFilter value "1g.5gb" must be a DNS label: must not contain dots

Profile names in partitionFilter are DNS labels — write 1g5gb (no dot). The dotted form is used only in CEL selectors and in PhysicalGPU capabilities.

What to do if an expected DeviceClass is missing from a GPUClass?

If a name you expected is absent from .status.deviceClassNames, the hardware most likely does not support that combination. Filters restrict what is published; they cannot create capabilities. Check what the card can actually do:

d8 k get physicalgpu <name> -o jsonpath='{.status.capabilities}' | jq

Also check .status.matchingPhysicalGPUCount — 0 means the selector matched nothing.

What to do if GPU allocation is stuck?

Every published device declares bindingConditions: ["Ready"] and bindingFailureConditions: ["BindingFailed"], so a failed preparation surfaces on the claim rather than leaving the Pod silently pending. Check the ResourceClaim:

d8 k -n <namespace> get resourceclaim <name> -o yaml

Preparation errors for VFIO claims are listed in Usage.

What to do if time-slicing was received instead of MPS sharing?

This is expected behavior and it is reported. MPS and time-slicing are advertised per node, based on whether the vendor’s binaries are present — the card model is never checked. So a claim asking for MPS is accepted even on hardware that cannot run it.

The mismatch surfaces during preparation. The module falls back to time-slicing and records an event. To view the events, run:

d8 k -n d8-nvidia-gpu get events --field-selector reason=MPSFallbackToTimeSlicing

The workload still gets sharing, just not the kind it requested: without MPS there is no per-client thread limit, so sharePercent no longer bounds compute the way it would under MPS. If the fallback itself fails, preparation stops with MPSPrepareFailed.

MPS requires compute capability 7.0 or higher. See Module capabilities by GPU vendor.

Why do DCGM pods disappear during a MIG change?

This is expected behavior. Before reconfiguring MIG, nvidia-adapter sets gpu.deploy.dcgm and gpu.deploy.dcgm-exporter to paused-for-mig-change so that DCGM stops holding the GPU. The DaemonSets return once the operation completes.

What to do if MIG reconfiguration or VFIO passthrough will not proceed?

The card must be free of running processes. Check on the node:

nvidia-smi

The Processes section must be empty. On nodes with a graphical environment the GPU is often held by Xorg, gnome-shell, gdm, sddm, or lightdm; they occupy GPU memory and block reconfiguration.

For VFIO specifically, the node also needs IOMMU enabled in BIOS and on the kernel command line, reflected by node.deckhouse.io/gpu-vfio-ready=true.

What to do if uninstalling or disabling the module hangs?

On uninstall, a pre-delete hook deletes every PhysicalGPU object and waits up to 600 seconds so gpu-controller can clear its finalizers before the CRD is removed. If the hook times out, look for leftover PhysicalGPU objects:

d8 k get physicalgpus -A

What to do if a GPU shows as Orphaned?

If d8 k get physicalgpus reports Management=Orphaned, a passthrough driver binding exists with no owning ResourceClaim and no deliberate opt-out — today that means a vfio-pci binding, the only one that classifies as Passthrough, left behind by a workload that vanished. Passthrough is required here, not merely typical: a card whose binding is in Compute mode with no claim on it is the ordinary free card, reads Unclaimed for as long as it stays free, and never becomes Orphaned. The classification only lands after the stray passthrough binding has persisted for two minutes: it is not a transient state.

Check the reason on the Managed condition:

d8 k get physicalgpu <name> -o jsonpath='{.status.conditions[?(@.type=="Managed")]}' | jq

Only Orphaned makes a device reclaim-eligible. The reason tells you which of the eight cases you are looking at, and each one fixes what the Management column reads:

• BackingAllocation → Managed. An allocated ResourceClaim references the card; it is in use.
• NoBackingAllocation → Unclaimed. Not a problem at all: the card is identified, its binding is a live compute one, and nobody holds it. Nothing needs doing.
• ExcludedReleasePending → Managed. An opt-out was declared while the binding is still alive, so the card is not yet reclaim-eligible.
• DeliberatelyExcluded → Excluded. The opt-out has settled and the binding was released.
• OrphanPending → Unknown. The grace period is still running.
• Orphaned → Orphaned. The stray passthrough binding held for the whole grace period — the case this section is about.
• NoLiveBinding → Unknown. There is no binding to manage: no driver is bound, the bound one could not be classified, or the binding is not published yet.
• Indeterminate → Unknown. Nothing could be read at all — no Node name or no PCI address, and either one missing is enough.

Three of them print Unknown in the column; of those only OrphanPending has a clock running.

To hand the device back, clear the stray passthrough binding and let discovery reclassify it, or request a reset as below.

What to do if a reclaim request is refused?

The gpu.deckhouse.io/reclaim annotation asks the controller to reset and rediscover a device. A request the controller cannot honour is refused rather than silently ignored, and the refusal is recorded on the Reclaim condition:

d8 k get physicalgpu <name> -o jsonpath='{.status.conditions[?(@.type=="Reclaim")]}' | jq

The condition is not a permanent record. The controller drops it once the Managed condition transitions past the refusal: the answer described a management state the card has since left, so it goes with it. Empty output therefore means one of two things — no reclaim was ever refused on this card, or the card has changed hands since it was. What outlives the condition is the ReclaimRefused event, until the apiserver sweeps it on --event-ttl (an hour by default), and after that the controller log, which records every refusal unconditionally:

d8 k get events --field-selector reason=ReclaimRefused

What the refusal almost always names is the Management column, not the obstacle behind it: reclaim is gated on managementState before anything else, and only Orphaned opens it. So read the column first, and let it decide what to do:

d8 k get physicalgpus

• Unclaimed. The refusal is correct and repeating the request will never succeed. An Unclaimed card is a live Compute binding nobody holds — a healthy free card, allocatable as it stands, with nothing to reclaim. Releasing a claim on such a card yields Unclaimed again, never Orphaned, so retrying after a release just loops.
• Managed. Either an allocated ResourceClaim holds the card, or it was opted out while its binding is still alive (reason ExcludedReleasePending on the Managed condition). The module never reclaims a card from under a live workload. Release the claim, then re-annotate only if the card actually reads Orphaned afterwards — that happens for a stray passthrough binding after its grace period, whereas a compute-bound card lands in Unclaimed and needs no reclaim at all.
• Excluded. The card was deliberately withdrawn with gpu.deckhouse.io/unmanaged and its binding is already released. There is nothing to reclaim; remove the annotation to put the card back under module management.
• Unknown with reason OrphanPending. The two-minute grace period is still running. The request is not refused but held, and reclaim starts on its own once the card becomes Orphaned. Wait rather than re-annotate.
• Unknown with reason NoLiveBinding or Indeterminate. There is no binding the module could reset, or the object has no Node name or no PCI address — either one missing is enough — so a claim cannot be correlated against the card at all. Re-annotating changes nothing. Restore the identity first: check that gpu-node-agent is running on the node and republishing status.nodeInfo.nodeName and status.pciInfo.address.

A card that does read Orphaned can still be refused. Past the column gate the controller re-checks occupancy and stays fail-closed, so a claim that turns up after all, or occupancy it cannot determine, refuses the request at that point. Those refusals are rare and mean the column and the claims disagreed — re-read both.

One terminal outcome on the Reclaim condition is not a refusal of your request at all: if the annotation is removed while a reclaim is in progress, the condition records the reclaim as abandoned. Re-annotate to start over.

Why does a passthrough request fail instead of entering Passthrough?

VFIO passthrough requires a bare-metal node. On a virtualised node the vendor adapter reports AdapterReady=False with reason VFIORequiresBareMetal, which derives to phase Failed — not Passthrough.

How do I take a GPU out of module management?

Annotate it with gpu.deckhouse.io/unmanaged. While the binding is still alive the device stays Managed with reason ExcludedReleasePending; once released it settles at Excluded.

How do I monitor GPUs?

DCGM Exporter is deployed automatically, its metrics are scraped by Prometheus, and dashboards are available in Grafana — cluster overview, per-node and per-namespace views, GPU inventory, and per-workload allocations. The module also exports its own metrics (gpu_inventory_*, gpu_bootstrap_*, gpu_resourceclaim_*). See Monitoring.

Are AMD or Intel GPUs supported?

Not for allocation. Adapters ship for NVIDIA and MetaX, and an adapter is what turns a discovered card into something a workload can claim.

What already works for other vendors is everything above the adapter. Discovery recognises AMD (1002) and Intel (8086) devices, so their cards appear as PhysicalGPU objects with PCI details filled in, and the node gets gpu.deckhouse.io/vendor=amd or =intel. Without an adapter those objects stay in phase Pending: nothing reads their capabilities, no ResourceSlice is published, and the scheduler never sees them.

d8 k get physicalgpus

NAME                   NODE       MODEL   PHASE     MANAGEMENT   STATUS                                AGE
worker-2-0-1002-7408   worker-2           Pending   Unknown      adapter not scheduled to a workload   3m

MANAGEMENT reads Unknown with reason NoLiveBinding. The amdgpu driver is not one the module classifies, so the card has no live compute binding and there is nothing for the module to manage. What puts the card here is the unclassified driver, not the missing adapter: an NVIDIA card whose adapter has not been scheduled is still bound to nvidia, still reads mode: Compute, and so reads Unclaimed — the same Pending phase, a different MANAGEMENT. Unclaimed is reserved for a card whose binding the module could manage and nobody has taken.

MODEL reads the normalized model slug from the gpu.deckhouse.io/device label and stays empty while the model cannot be resolved; the PCI vendor and device IDs are still in the object name, and the PCI address is in the wide output. STATUS spells out why a card is not Ready, and it is filled by gpu-controller: here it repeats the message of the AdapterReady=False/NotScheduled condition that put the card in Pending. The vendor, the PCI address and the health of the device also live in the wide output (d8 k get physicalgpus -o wide).

Adding a vendor does not require changes to the CRD or to any core component: it needs a container implementing the adapter contract, a PCI vendor ID in the gpu-supported-vendors ConfigMap, and an adapter DaemonSet.

Device Plugin mode

The following applies only to Device Plugin mode (dra.enabled: false).

How do I verify the Device Plugin mode components?

NVIDIA Pods in d8-nvidia-gpu:

d8 k -n d8-nvidia-gpu get pod

Expected healthy output (example):

NAME                                  READY   STATUS    RESTARTS   AGE
gpu-feature-discovery-80ceb7d-r842q   2/2     Running   0          2m53s
nvidia-dcgm-exporter-w9v9h            1/1     Running   0          2m53s
nvidia-dcgm-njqqb                     1/1     Running   0          2m53s
nvidia-device-plugin-80ceb7d-8xt8g    2/2     Running   0          2m53s

NFD Pods in d8-nvidia-gpu:

d8 k -n d8-nvidia-gpu get pods | egrep '^(NAME|node-feature-discovery)'

Expected healthy output (example):

NAME                                             READY   STATUS      RESTARTS       AGE
node-feature-discovery-gc-6d845765df-45vpj       1/1     Running     0              3m6s
node-feature-discovery-master-74696fd9d5-wkjk4   1/1     Running     0              3m6s
node-feature-discovery-worker-5f4kv              1/1     Running     0              3m8s

What to do if there is an Incompatible strategy detected auto error in the device plugin or GFD logs?

Errors such as Incompatible strategy detected auto, failed to create resource manager: unsupported strategy auto, or invalid device discovery strategy mean the component cannot detect the NVML platform inside the container — usually libnvidia-ml.so.* is unavailable because the NVIDIA Container Toolkit runtime is not in use.

Check (and fix if needed) the following:

1. nvidia-smi works on the node.
2. NVIDIA Container Toolkit is installed (/usr/bin/nvidia-container-runtime exists).
3. containerd is configured to use the nvidia runtime on GPU nodes. The module does this after the driver and toolkit are installed and containerd is restarted or the node is rebooted.

Afterwards, recreate the nvidia-device-plugin-* and gpu-feature-discovery-* pods in the d8-nvidia-gpu namespace.

How do I view available MIG profiles?

Predefined profiles live in the mig-parted-config ConfigMap:

d8 k -n d8-nvidia-gpu get cm mig-parted-config -o json | jq -r '.data["config.yaml"]'

The mig-configs: section lists GPU models by PCI ID and the profiles each card supports, such as all-1g.5gb, all-2g.10gb, and all-balanced. Set the chosen name in spec.gpu.mig.partedConfig.

How do I define a custom MIG profile per GPU on a node?

Use partedConfig: custom and describe the partitioning per GPU index:

gpu:
  sharing: MIG
  mig:
    partedConfig: custom
    customConfigs:
      - index: 0
        slices:
          - profile: "1g.10gb"
            count: 7
      - index: 1
        slices:
          - profile: "2g.20gb"
            count: 3

The module then generates a unique MIG config name for the NodeGroup and sets it in the nvidia.com/mig.config label, renders mig-enabled: true with the declared slices for the listed GPUs, and renders mig-enabled: false for every unlisted index so the remaining cards stay in full mode.

What to do if a MIG profile does not activate?

If a MIG profile does not activate, do the following:

1. Check the GPU model. MIG is supported on H100, A100, and A30, and not on V100 or T4. See the NVIDIA profile tables.

2. Make sure the GPU is not in use. See What to do if MIG reconfiguration or VFIO passthrough will not proceed.

3. Check the NodeGroup configuration.

4. Wait for nvidia-mig-manager to drain the node and reconfigure the GPU. This can take several minutes, during which the node carries the mig-reconfigure taint.

5. Track progress through the nvidia.com/mig.config.state label: pending, rebooting, success, or failed.

6. If nvidia.com/mig-* resources still do not appear, run:

   d8 k -n d8-nvidia-gpu logs daemonset/nvidia-mig-manager
   nvidia-smi -L