Administration Guide

This guide is for a cluster administrator. You enable the module, decide what its orders are allowed to do, and watch it. Ordering inference itself is the namespace user’s task — see User guide.

The division of labour is the point of the module and worth stating once: an order names a class and a model and nothing else. Everything a user cannot express — which accelerator, which share, which runtime, which inference API contract, how many replicas — is either assigned by the platform or bounded by the class you write. The class is the whole of your control surface.

Requirements

• Deckhouse Kubernetes Platform >= 1.75.0;
• Kubernetes >= 1.34. The floor matches the gpu and ai-models modules on the same contour; the core Dynamic Resource Allocation API resource.k8s.io/v1 is already present on that version;
• editions fe and ee;
• accelerators exposed through Dynamic Resource Allocation, with device classes the platform can select on. Placement is planned against real hardware facts, not requested by hand.

Enable the module

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: ai-inference
spec:
  enabled: true
  version: 1

The module has exactly one setting, and it is about integration with the ai-models catalog module:

spec:
  settings:
    catalog:
      mode: Enabled   # or None

Enabled — the default — keeps catalog lookup and the permissions for inter-module access. None disables the catalog clients and those permissions; orders that name a direct Hugging Face source keep working. Turn it off when your orders never use the catalog and ai-models is not installed. The full reference is Configuration.

The module runs in namespace d8-ai-inference and ships two components: the controller, which reconciles orders into workloads, and the resource planner, which decides placement.

Author an inference service class

InferenceServiceClass is cluster-scoped. It is the only place where policy lives, and an order cannot override any of it.

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: llm-chat-external
spec:
  exposurePolicy:
    type: External
    authentication: Token
    https:
      mode: CertManager
      certManager:
        clusterIssuerName: letsencrypt
  admissionPolicy:
    allowedNamespaces:
      - support
      - analytics
  modelPolicy:
    allowedEndpointTypes:
      - Chat
  scalingPolicy:
    minReplicas: 1
    maxReplicas: 4
  acceleratorPolicy:
    allowedSharingModes:
      - Shared
      - Dedicated
    allowedPlacementTypes:
      - WholeDevice
      - Partition
    maxAcceleratorCount: 1
  updatePolicy:
    strategy: RollingUpdate

What each block decides:

modelStorageClassName reaches new volumes only. A volume does not move to another StorageClass, and the server refuses every change to the claim templates of a standing workload — so an order whose volume already exists keeps the class it was made on, whatever the class now says. Nothing breaks and nothing stops: the platform keeps the standing claim and records the divergence on the order as a SettledRegionKept event. To move an existing order onto the new class, delete its workload and the claim behind it; the artefact is then downloaded again.

Three of these deserve a warning, because their default is a decision and not an absence:

modelPolicy.allowedEndpointTypes is required and must be non-empty, enforced by a validation rule on the resource. The contract of an order is picked from this list; an order names none. A class without the list would leave its orders with no contract at all, so the API server refuses it on write.

The order of that list is part of the decision. The platform takes the first element of the effective set, where the effective set is your list narrowed by whatever the model source says the model can serve. On a source that carries facts about the model — the catalog — an embedding model gets the embedding contract. On a source that carries no facts, the whole list stands and the first element wins: an embedding model ordered from a bare Hugging Face reference would be served as chat. If that matters to your users, declare a class whose list names one contract.

A class without scalingPolicy gives its orders one replica and no autoscaler at all. That is a capacity decision, and the module does not make it on your behalf.

A class without acceleratorPolicy gives every order a whole accelerator. Two orders that would each fit in half a card then occupy two cards, and the co-placement the module exists to offer never happens on that class.

The bundled class

The module ships a cluster-scoped class named default-llm. It allows all three contracts with Chat first, permits both sharing modes and both placement types with one device per replica, names no share bounds — so the plan assigns the smallest share that admits the model’s memory — and bounds replicas at one to two. It exposes External with authentication: Token.

It is a starting point, not a policy for your cluster: it names no admissionPolicy.allowedNamespaces choice of yours, and its replica ceiling of two is the smallest number at which an autoscaler can exist at all.

Grant access

The module ships cluster roles; you bind them.

Denied to humans on purpose: inferenceserviceclasses/planner, every */status subresource, Secrets, and bind / escalate / impersonate. The PrivilegedUser, Admin and ClusterAdmin deltas are empty — those levels gain no extra action here.

Note what a namespace user cannot do: read a class. That is why status.model.endpointType exists on an order — it is the user’s only window on a decision the class made.

Watch the module

Two alerts ship with the module, both about capacity moving between orders:

• D8AIInferenceCapacityPreempted — capacity was preempted for a recipient order on a device class. Something grew at another order’s expense;
• D8AIInferenceDonorHeldAtLoweredBound — a donor order is being held at a lowered replica bound. It gave capacity up and has not got it back.

Two Grafana dashboards ship as well, both about the request runtime: performance statistics and query statistics.

An order in phase Degraded is the state to notice: its endpoint answers, but it cannot grow. The reason is usually ScaleUpBlocked — preemption exhausted the capacity the autoscaler needed.

Operational checks

# which classes exist and whether they validated
kubectl get inferenceserviceclass

# orders across the cluster with the class each one uses
kubectl get inferenceservice -A

# why a class is not Ready
kubectl get inferenceserviceclass llm-chat-external \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.reason}{"\n"}{end}'

A class publishes two conditions, Validated and Ready. An order that reports PlacementUndecidable is pointing at your class rather than at the cluster’s capacity: it means the placement could not be judged at all, so the selector expression of the class needs an eye. Compare that with ResourceClaimExhausted, which means the search did happen and every allowed device class was tried — that one is about capacity.

Upgrading the module

An upgrade of the module can bring a new runtime release with it, and what happens to the orders already in the cluster is worth knowing before you press the button.

Replicas that are serving keep serving, on the release they were started with. The description of their workload is updated on the first reconcile after the upgrade, so a replica raised from then on — because the autoscaler widened the set, or because one was replaced after a node went away — comes up on the shipped release. The ones already running are left alone: the alternative is a restart, and a restart means loading the model again, which takes minutes per replica.

The order says where it stands. While the two differ, the order carries RuntimeCurrent=False with reason ReplicasHeldOnPreviousRuntime, and the message names both releases and how many replicas reached the shipped one:

kubectl get inferenceservice support-llm \
  -o jsonpath='{range .status.conditions[?(@.type=="RuntimeCurrent")]}{.reason}: {.message}{"\n"}{end}'

Two ways to finish the transition. Editing the order moves every replica the way the class asks for — that is a rolling restart with the downtime it implies. Deleting a replica moves that one only:

kubectl delete pod support-llm-0 -n my-namespace

While the set is mixed, the replicas answer differently. The launch parameters come from the recipe of the release each replica was started with, and the recipe sets the context length and the concurrency limit. A request near those limits can therefore be served by one replica and refused by another until the transition finishes. If that matters for your workload, edit the order right after the upgrade and take the downtime at a time you choose.

When the transition cannot be lazy. If the recomputed plan moves the placement of the order — another device class, a different share of a device, a larger model volume — the replicas cannot be kept: the objects holding the devices are the ones that have to change. The order then reports RuntimeCurrent=False with reason WorkloadRebuilding, goes through Pending, and comes back on the new release.

Disable the module

Disabling asks for confirmation, and the warning is worth reading rather than clicking through. Disabling stops the controller and the resource planner, and with them every order workload they keep running: the request runtime, its service and its published endpoint.

Clean up first, in this order:

1. delete every InferenceService in every namespace;
2. delete the InferenceServiceClass objects those namespaces reference.

Disabling without that cleanup leaves order objects in the cluster with nothing reconciling them.

Where to look next

• Configuration — the settings reference;
• Custom Resources — the full field reference of both kinds;
• Examples — class and order manifests to copy;
• FAQ — answers organised by refusal reason.