The module lifecycle stage: General Availability

The module has requirements for installation

This example covers running a workload (Pod) on a GPU when the module is in DRA mode. In this mode the module discovers GPUs on all nodes by itself, and the sharing method is chosen per workload rather than per node group.

To run a workload on a GPU, do the following:

  1. Enable the module in DRA mode.
  2. Install the NVIDIA driver on the node.
  3. Confirm that the module discovered the card.
  4. Publish a GPU pool.
  5. Run a workload on the GPU.

To run GPU workloads, the cluster must meet the requirements.

The examples below assume that the node where the workload runs is on Ubuntu, and that nothing GPU-related is configured on it yet.

Enabling the module

Before enabling the module in DRA mode, make sure the cluster meets the prerequisites for DRA mode.

Enable the module in DRA mode. To do that, apply a ModuleConfig:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: gpu
spec:
  enabled: true
  version: 1
  settings:
    dra:
      enabled: true

Wait for the module to become ready. To check readiness, use:

d8 k get module gpu -o jsonpath='{.status.phase}'

Example output when the module is ready:

Ready

Installing the NVIDIA driver on the node

Create a NodeGroupConfiguration resource that installs the driver and the NVIDIA Container Toolkit. Replace gpu with the name of your NodeGroup:

apiVersion: deckhouse.io/v1alpha1
kind: NodeGroupConfiguration
metadata:
  name: install-cuda.sh
spec:
  bundles:
    - ubuntu-lts
  content: |
    #!/bin/bash
    if [ ! -f "/etc/apt/sources.list.d/nvidia-container-toolkit.list" ]; then
      distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
      curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo apt-key add -
      curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
    fi
    bb-apt-install nvidia-container-toolkit nvidia-driver-535-server
    nvidia-ctk config --set nvidia-container-runtime.log-level=error --in-place
  nodeGroups:
    - gpu
  weight: 30

Reboot the node so the driver loads. Then connect to the node and check the card:

nvidia-smi

Example output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.144.03             Driver Version: 550.144.03     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla T4                       Off |   00000000:01:00.0 Off |                    0 |
| N/A   61C    P8             17W /   70W |       1MiB /  15360MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

The Processes section at the bottom must be empty. If a process is holding the GPU, stop it before continuing.

Confirm that the module discovered the card

The module finds the GPU (card) and creates a PhysicalGPU object for it. To list all PhysicalGPU objects, use:

d8 k get physicalgpus

Example output:

NAME                            NODE                VENDOR   DEVICE   MODEL      PHASE   MODE      HEALTHY
dev-ee-ubuntu24-0-0-10de-1eb8   dev-ee-ubuntu24-0   nvidia   1eb8     tesla-t4   Ready   Compute   True

Wait until PHASE is Ready and HEALTHY is True. Then check that the card is offered to the scheduler:

d8 k get resourceslices

Example output:

NAME                              NODE                DRIVER               POOL                AGE
dev-ee-ubuntu24-0-gpu.deckhouse   dev-ee-ubuntu24-0   gpu.deckhouse.io     dev-ee-ubuntu24-0   30s

Publishing a GPU pool

Create a GPUClass covering every GPU in the cluster:

apiVersion: gpu.deckhouse.io/v1alpha1
kind: GPUClass
metadata:
  name: gpu-any
spec: {}

See which names it created:

d8 k get gpuclass gpu-any -o jsonpath='{.status.deviceClassNames}' | jq

Example output:

[
  "gpu-any",
  "gpu-any-mps-percent",
  "gpu-any-ts-percent"
]

The first name, gpu-any, allocates a whole card. You will need it in the step that runs a workload on the GPU.

Running a workload on the GPU

Create and apply a Job resource. In the workload spec (containers), point at the GPU pool you created earlier:

apiVersion: batch/v1
kind: Job
metadata:
  name: gpu-hello
  namespace: default
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: probe
          image: nvidia/cuda:12.8.0-devel-ubuntu24.04
          command: ["nvidia-smi", "-L"]
          resources:
            requests:
              gpu.deckhouse.io/gpu-any: 1
            limits:
              gpu.deckhouse.io/gpu-any: 1

Check the logs of the created resource:

d8 k logs job/gpu-hello

Example output with information about the GPU the Pod received:

GPU 0: Tesla T4 (UUID: GPU-1c8f4a2e-6b3d-4f9a-8e15-7d2c9b0a4f61)