The module lifecycle stage: General Availability

The module has requirements for installation

This section provides ready-to-apply manifests for typical administrator and user tasks.

GPU nodes are often tainted (for example node-role=gpu:NoSchedule) so that regular workloads do not land there. To allow scheduling of regular workloads onto such nodes, add matching tolerations and a nodeSelector to the examples below.

Custom taint keys must be allowlisted in ModuleConfig global under .spec.settings.modules.placement.customTolerationKeys, otherwise creating the NodeGroup is rejected:

it is forbidden to create a NodeGroup resource with taints not specified in ModuleConfig "global"
in the array .spec.settings.modules.placement.customTolerationKeys, add: node-role to customTolerationKeys

Administrator: publishing GPU pools

All GPUs of one model

apiVersion: gpu.deckhouse.io/v1alpha1
kind: GPUClass
metadata:
  name: a100
spec:
  selector:
    matchLabels:
      gpu.deckhouse.io/device: a100-sxm4-40gb

Small MIG partitions only

apiVersion: gpu.deckhouse.io/v1alpha1
kind: GPUClass
metadata:
  name: a100-small
spec:
  selector:
    matchLabels:
      gpu.deckhouse.io/device: a100-sxm4-40gb
  partitionFilter:
    allow:
      - 1g5gb
      - 2g10gb

Profile names in the filter are DNS labels: 1g5gb, not 1g.5gb.

A pool that cannot be shared

apiVersion: gpu.deckhouse.io/v1alpha1
kind: GPUClass
metadata:
  name: a100-exclusive
spec:
  selector:
    matchLabels:
      gpu.deckhouse.io/device: a100-sxm4-40gb
  sharingFilter:
    deny:
      - mps
      - ts

Cards on a specific node

apiVersion: gpu.deckhouse.io/v1alpha1
kind: GPUClass
metadata:
  name: training-rig
spec:
  selector:
    matchLabels:
      gpu.deckhouse.io/node: worker-gpu-1

User: requesting a published class

Look up what is available, then request it by name.

To list all GPUClass objects in the cluster, use:

d8 k get gpuclasses

To list all DeviceClass objects that the GPU controller automatically created from a GPUClass, use:

d8 k get gpuclass a100 -o jsonpath='{.status.deviceClassNames}' | jq

Requesting a whole GPU

apiVersion: batch/v1
kind: Job
metadata:
  name: train
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: train
          image: nvidia/cuda:12.8.0-devel-ubuntu24.04
          command: ["nvidia-smi", "-L"]
          resources:
            requests:
              gpu.deckhouse.io/a100: 1
            limits:
              gpu.deckhouse.io/a100: 1

Requesting a MIG partition

      resources:
        requests:
          gpu.deckhouse.io/a100-1g5gb: 1
        limits:
          gpu.deckhouse.io/a100-1g5gb: 1

Requesting a quarter of a shared GPU

      resources:
        requests:
          gpu.deckhouse.io/a100-mps-percent: 25
        limits:
          gpu.deckhouse.io/a100-mps-percent: 25

User: writing the claim yourself

Use this when the published classes do not cover your needs.

Requesting a whole GPU exclusively

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: nvidia-whole-gpu
spec:
  selectors:
    - cel:
        expression: |
          device.attributes["gpu.deckhouse.io"].vendor == "nvidia" &&
          device.attributes["gpu.deckhouse.io"].computeAPI == "CUDA" &&
          has(device.attributes["gpu.deckhouse.io"].deviceType) &&
          device.attributes["gpu.deckhouse.io"].deviceType == "physical" &&
          !has(device.attributes["gpu.deckhouse.io"].sharingStrategy)
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: whole-gpu
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: nvidia-whole-gpu
            count: 1
---
apiVersion: batch/v1
kind: Job
metadata:
  name: whole-gpu
spec:
  backoffLimit: 0
  template:
    spec:
      restartPolicy: Never
      resourceClaims:
        - name: gpu
          resourceClaimTemplateName: whole-gpu
      containers:
        - name: probe
          image: nvidia/cuda:12.8.0-devel-ubuntu24.04
          command: ["nvidia-smi", "-L"]
          resources:
            claims:
              - name: gpu

Requesting a specific MIG profile

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: nvidia-mig-2g10gb
spec:
  selectors:
    - cel:
        expression: |
          device.attributes["gpu.deckhouse.io"].vendor == "nvidia" &&
          device.attributes["gpu.deckhouse.io"].computeAPI == "CUDA" &&
          has(device.attributes["gpu.deckhouse.io"].deviceType) &&
          device.attributes["gpu.deckhouse.io"].deviceType == "partition" &&
          has(device.attributes["gpu.deckhouse.io"].partitionTechnology) &&
          device.attributes["gpu.deckhouse.io"].partitionTechnology == "MIG" &&
          has(device.attributes["gpu.deckhouse.io"].partitionProfile) &&
          device.attributes["gpu.deckhouse.io"].partitionProfile == "2g.10gb"

Profile names in a CEL selector are the hardware names, with dots.

Requesting a specific card by PCI address

    - cel:
        expression: |
          device.attributes["gpu.deckhouse.io"].deviceType == "physical" &&
          device.attributes["gpu.deckhouse.io"].pciAddress == "00000000:d5:00.0"

Requesting a quarter of a GPU via MPS

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: nvidia-gpu-mps
spec:
  selectors:
    - cel:
        expression: |
          device.attributes["gpu.deckhouse.io"].vendor == "nvidia" &&
          device.attributes["gpu.deckhouse.io"].computeAPI == "CUDA" &&
          has(device.attributes["gpu.deckhouse.io"].deviceType) &&
          device.attributes["gpu.deckhouse.io"].deviceType == "physical" &&
          has(device.attributes["gpu.deckhouse.io"].sharingStrategy) &&
          device.attributes["gpu.deckhouse.io"].sharingStrategy == "mps"
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: gpu-mps-quarter
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: nvidia-gpu-mps
            count: 1
            capacity:
              requests:
                sharePercent: "25"
                gpu.deckhouse.io/memory: 1Gi
      config:
        - requests: ["gpu"]
          opaque:
            driver: gpu.deckhouse.io
            parameters:
              apiVersion: resource.gpu.deckhouse.io/v1alpha1
              kind: GpuConfig
              sharing:
                strategy: MPS
                mpsConfig:
                  defaultActiveThreadPercentage: 25

MPS on top of a MIG partition

Same idea, but with MigDeviceConfig and a partition selector. On a partition, sharePercent alone is enough:

      requests:
        - name: gpu
          exactly:
            deviceClassName: nvidia-mig-2g10gb-mps
            count: 1
            capacity:
              requests:
                sharePercent: "25"
      config:
        - requests: ["gpu"]
          opaque:
            driver: gpu.deckhouse.io
            parameters:
              apiVersion: resource.gpu.deckhouse.io/v1alpha1
              kind: MigDeviceConfig
              sharing:
                strategy: MPS
                mpsConfig:
                  defaultActiveThreadPercentage: 25

Any share, without specifying a strategy

The selector must stay permissive — do not exclude sharing variants:

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: nvidia-gpu-any
spec:
  selectors:
    - cel:
        expression: |
          device.attributes["gpu.deckhouse.io"].vendor == "nvidia" &&
          device.attributes["gpu.deckhouse.io"].computeAPI == "CUDA" &&
          has(device.attributes["gpu.deckhouse.io"].deviceType) &&
          device.attributes["gpu.deckhouse.io"].deviceType == "physical"
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: gpu-share
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: nvidia-gpu-any
            count: 1
            capacity:
              requests:
                sharePercent: "25"
                gpu.deckhouse.io/memory: 1Gi

VFIO passthrough

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: h100-vfio
spec:
  selectors:
    - cel:
        expression: |
          device.attributes["gpu.deckhouse.io"].vendor == "nvidia" &&
          has(device.attributes["gpu.deckhouse.io"].productName) &&
          device.attributes["gpu.deckhouse.io"].productName.lowerAscii().matches("^.*h100.*$") &&
          has(device.attributes["gpu.deckhouse.io"].deviceType) &&
          device.attributes["gpu.deckhouse.io"].deviceType == "physical" &&
          has(device.attributes["gpu.deckhouse.io"].bareMetal) &&
          device.attributes["gpu.deckhouse.io"].bareMetal == true
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: h100-vfio
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: h100-vfio
            allocationMode: ExactCount
            count: 1
      config:
        - requests: ["gpu"]
          opaque:
            driver: gpu.deckhouse.io
            parameters:
              apiVersion: resource.gpu.deckhouse.io/v1alpha1
              kind: VfioDeviceConfig
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: h100-vfio
spec:
  replicas: 1
  selector:
    matchLabels:
      app: h100-vfio
  template:
    metadata:
      labels:
        app: h100-vfio
    spec:
      nodeSelector:
        node.deckhouse.io/gpu-vfio-ready: "true"
      resourceClaims:
        - name: gpu
          resourceClaimTemplateName: h100-vfio
      containers:
        - name: vfio-consumer
          image: busybox:1.36
          command: ["/bin/sh", "-c", "ls -l /dev/vfio && sleep infinity"]
          resources:
            claims:
              - name: gpu

Inside a passthrough container there is no CUDA and no nvidia-smi — only VFIO device nodes.

Manifests for Device Plugin mode

These examples apply to Device Plugin mode (with dra.enabled: false). Sharing is configured per NodeGroup through spec.gpu, and workloads request nvidia.com/gpu.

Exclusive

apiVersion: deckhouse.io/v1
kind: NodeGroup
metadata:
  name: gpu-exclusive
spec:
  nodeType: Static # Or CloudStatic/CloudEphemeral as needed.
  gpu:
    sharing: Exclusive
  nodeTemplate:
    labels:
      node-role/gpu: ""
    taints:
      - key: node-role
        value: gpu
        effect: NoSchedule

TimeSlicing

spec:
  gpu:
    sharing: TimeSlicing
    timeSlicing:
      partitionCount: 4

MIG

spec:
  gpu:
    sharing: MIG
    mig:
      partedConfig: all-1g.5gb

For per-index custom partitioning use partedConfig: custom with customConfigs. See Diagnostics.

Verification run

apiVersion: batch/v1
kind: Job
metadata:
  name: cuda-vectoradd
spec:
  template:
    spec:
      restartPolicy: OnFailure
      nodeSelector:
        node-role/gpu: ""
      tolerations:
        - key: node-role
          value: gpu
          effect: NoSchedule
      containers:
        - name: cuda-vectoradd
          image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1-ubuntu20.04
          resources:
            limits:
              nvidia.com/gpu: 1