The module lifecycle stage: Experimental

The module has requirements for installation

Use this page as a manifest catalog. When to choose each shape and how to diagnose a failure is explained in the User guide and the Administration guide; this page only carries the manifests.

Manifests exercised against a live stand live in examples/ in the repository.

Minimal ModuleConfig

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: ai-inference
spec:
  enabled: true
  version: 1

ModuleConfig without the catalog

For a cluster where orders name direct model sources and ai-models is not installed.

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: ai-inference
spec:
  enabled: true
  version: 1
  settings:
    catalog:
      mode: None

Minimal order

A class reference and a model. This is the only shape an order has.

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: support-llm
  namespace: support
spec:
  inferenceServiceClassName: default-llm
  model:
    ref:
      name: Qwen/Qwen2.5-32B-Instruct-AWQ
    src: HuggingFace

Order of a gated Hugging Face model

The Secret lives in the order’s own namespace. key is optional; the default key is token.

apiVersion: v1
kind: Secret
metadata:
  name: hf-token
  namespace: support
type: Opaque
stringData:
  token: <HUGGING_FACE_TOKEN>
---
apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: llama-chat
  namespace: support
spec:
  inferenceServiceClassName: default-llm
  model:
    ref:
      name: meta-llama/Llama-3.1-8B
    src: HuggingFace
    authSecretRef:
      name: hf-token
      key: token

Order of a catalog model

kind selects between a namespaced Model and a cluster-scoped ClusterModel. A namespace in ref is allowed for Model only.

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: catalog-chat
  namespace: support
spec:
  inferenceServiceClassName: default-llm
  model:
    ref:
      kind: ClusterModel
      name: qwen2-5-8b-instruct-awq
    src: ai-models

Class published inside the cluster

No ingress, no certificate, no token.

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: llm-internal
spec:
  admissionPolicy:
    allowedNamespaces:
      - support
  modelPolicy:
    allowedEndpointTypes:
      - Chat
  exposurePolicy:
    type: ClusterLocal
    authentication: None
  updatePolicy:
    strategy: RollingUpdate

This class names no scalingPolicy and no acceleratorPolicy, so its orders get one replica and a whole accelerator each. Both are decisions — see the administration guide.

Class published outside the cluster

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: llm-chat-external
spec:
  admissionPolicy:
    allowedNamespaces:
      - support
      - analytics
  modelPolicy:
    allowedEndpointTypes:
      - Chat
  exposurePolicy:
    type: External
    authentication: Token
    https:
      mode: CertManager
      certManager:
        clusterIssuerName: letsencrypt
  scalingPolicy:
    minReplicas: 1
    maxReplicas: 4
  acceleratorPolicy:
    allowedSharingModes:
      - Shared
      - Dedicated
    allowedPlacementTypes:
      - WholeDevice
      - Partition
    maxAcceleratorCount: 1
  updatePolicy:
    strategy: RollingUpdate

Class bounding every policy it has

Every block filled in, including a named device class and a named priority class. This is the shape to copy when a class must not spill outside one generation of hardware.

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: llm-strict
spec:
  admissionPolicy:
    allowedNamespaces:
      - support
  modelPolicy:
    allowedEndpointTypes:
      - Chat
    maxParameterCount: 1B
  acceleratorPolicy:
    allowedSharingModes:
      - Shared
      - Dedicated
    allowedPlacementTypes:
      - Partition
      - WholeDevice
    allowedDeviceClasses:
      - nvidia-t4-mps-physical
    maxAcceleratorCount: 1
  exposurePolicy:
    type: External
    authentication: Token
    https:
      mode: CertManager
      certManager:
        clusterIssuerName: selfsigned
  scalingPolicy:
    allowedPriorityClassNames:
      - inference-normal
    minReplicas: 1
    maxReplicas: 4
  updatePolicy:
    strategy: RollingUpdate

Class for embeddings only

One contract in the list, so no order of this class can be served as chat.

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: embeddings-only
spec:
  admissionPolicy:
    allowedNamespaces:
      - analytics
  modelPolicy:
    allowedEndpointTypes:
      - Embeddings
  exposurePolicy:
    type: ClusterLocal
    authentication: None
  acceleratorPolicy:
    allowedSharingModes:
      - Shared
    allowedPlacementTypes:
      - WholeDevice
      - Partition
    maxAcceleratorCount: 1
  updatePolicy:
    strategy: RollingUpdate

Priority classes for preemption

Capacity moves between orders by priority. The class admits a priority class by name through scalingPolicy.allowedPriorityClassNames.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: inference-low
value: 100
globalDefault: false
description: Orders that yield capacity first.
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: inference-normal
value: 500
globalDefault: false
description: Everyday orders.
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: inference-high
value: 1000
globalDefault: false
description: Orders that take capacity from the others.

Read the result of an order

kubectl get inferenceservice support-llm -n support -o yaml

# just the answers
kubectl get inferenceservice support-llm -n support -o jsonpath='{.status.phase}{"\n"}'
kubectl get inferenceservice support-llm -n support -o jsonpath='{.status.endpoint.url}{"\n"}'
kubectl get inferenceservice support-llm -n support -o jsonpath='{.status.model.endpointType}{"\n"}'

# conditions with their reasons
kubectl get inferenceservice support-llm -n support \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.reason}{"\n"}{end}'

Call the endpoint

URL=$(kubectl get inferenceservice support-llm -n support -o jsonpath='{.status.endpoint.url}')
SECRET=$(kubectl get inferenceservice support-llm -n support -o jsonpath='{.status.authSecretName}')
TOKEN=$(kubectl get secret "$SECRET" -n support -o jsonpath='{.data.token}' | base64 -d)

curl -sS "$URL/v1/models" -H "Authorization: Bearer $TOKEN"

curl -sS "$URL/v1/chat/completions" \
  -H "Authorization: Bearer $TOKEN" \
  --json '{"model":"Qwen/Qwen2.5-32B-Instruct-AWQ","messages":[{"role":"user","content":"ping"}]}'