Phase 2.1 walkthrough

This page is the phase history of the module: how each capability arrived, in the order it arrived. It is kept because the reasoning behind a capability is worth reading, and because a cluster on an earlier phase still matches an earlier section.

It is not the place to learn today’s behaviour. For that:

• User guide — order inference, read the status, find out why an order waits;
• Administration guide — enable the module, author classes, grant access, watch, disable;
• Examples — the manifests, without the narrative.

The ai-inference controller deploys the order workload itself: a StatefulSet with the request runtime, a Service, exposure, a device claim and the rest of the order objects. No delivery package is involved.

When the module is enabled, Helm creates cluster-scoped default-llm (External, Token) with the standard module: ai-inference label, replica bounds and an accelerator policy that permits a shared whole device. Use it directly: the order asks for no accelerator, and the platform assigns the device, the count and the share of it.

1. InferenceServiceClass

Bundled class (module)

After module install the cluster already has:

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: default-llm
spec:
  modelPolicy:
    allowedEndpointTypes:
      - Chat
      - Embeddings
      - Rerank
  exposurePolicy:
    type: External
    authentication: Token
    https:
      mode: CertManager
      certManager:
        clusterIssuerName: selfsigned

Expected: status.phase: Ready.

allowedEndpointTypes is what lets an order name only a class and a model: an order names no contract of its own, and the platform picks one from this list. The bundled class permits every contract the module supports — Chat, Embeddings and Rerank — and the ORDER of the list matters, because the platform takes the first element of the effective set.

The effective set is this list narrowed by whatever the source of the model says the model can serve. On the ai-models catalog path that narrowing happens: an embedding model is served as an embedding model on this very class. On the direct Hugging Face path there are no such facts, so the whole list stands and the FIRST contract wins — Chat. An order of an embedding model on that path is served as chat, and nothing refuses it. If that is your case, declare your own InferenceServiceClass whose list names the single contract you need.

Additional administrator class (optional)

spec.acceleratorPolicy.allowedDeviceClasses is optional — it sets the allow-list of DeviceClass names for orders on that class, and it is the only allow-list there is: an order names no device class of its own.

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: llm-chat-shared
spec:
  acceleratorPolicy:
    allowedDeviceClasses:
      - nvidia-h100-mps-mig
    allowedSharingModes:
      - Shared
    allowedPlacementTypes:
      - WholeDevice
    maxAcceleratorCount: 1
    minSharePercent: 25
    maxSharePercent: 100
  exposurePolicy:
    type: External
    authentication: Token

Expected: status.phase: Ready, conditions Validated=True, Ready=True.

2. InferenceService (namespace user)

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: support-llm
  namespace: support
spec:
  inferenceServiceClassName: default-llm
  model:
    ref:
      name: Qwen/Qwen2.5-32B-Instruct-AWQ
    src: HuggingFace

An order is a class reference and a model, and nothing else. Its spec has no other fields.

The inference API contract is picked by the platform out of the allow-list of the class intersected with the facts of the model: the only allowed contract, or the first of several. The chosen value shows up in status.model.endpointType — the one window the owner of an order has on that choice, because the class is not theirs to read. An order that needs a different contract needs a different class.

The name of the runtime and every launch parameter come from the recipe of the launch plan, compiled against the hardware the plan chose. MM catalog keys (limitMmPerPrompt and similar) live there too. A different value is a different recipe or a different class — an administrator’s decision.

A class must declare a non-empty allow-list of API contracts. A class without one is refused on write: the contract is picked from that list, an order names none, and a class without it would leave its orders with no contract at all.

An order that still carries a runtime block is rejected by a client asking for strict field validation, and silently pruned by one that does not.

Before persisting an object, the API server validates the model source union: it rejects model namespace fields on HuggingFace and ClusterModel references, ClusterLocal classes with https, and certificate branches that do not match https.mode.

3. Controller behavior

1. F2-07 validation (class exists, is Ready and declares its allow-list of API contracts; model.src=HuggingFace).
2. Build the order workload objects and apply them in namespace support; every object is named after the order.
3. Compute settings.domain (for example support-llm.services.company.com) and publish status.endpoint.
4. When authentication: Token, create Secret {name}-auth and set status.authSecretName.
5. HTTP probe against the Inference API; set phase: Ready only after success.
6. After Ready, periodic healthcheck; sustained API errors → phase: Failed, reason ServiceUnhealthy (bootstrap before first Ready stays Pending, reason HealthCheckFailed).

4. Verify the result

Reading the status of a ready order is not phase history — it is how the module behaves today, and it is described once, in the User guide. The per-phase sections below say only what each phase ADDED to that status.

5. Hugging Face token (optional)

For gated models, create a Secret in the order namespace and reference it in spec.model.authSecretRef:

spec:
  model:
    ref:
      name: meta-llama/Llama-3.1-8B
    src: HuggingFace
    authSecretRef:
      name: hf-token
      key: token   # optional; the default key is "token"

The controller validates the Secret during F2-07 and mounts it into the model artifact fetcher of the order workload as HF_TOKEN.

RBAC

Intentionally denied to humans: inferenceserviceclasses/planner, */status, Secrets, bind / escalate / impersonate. PrivilegedUser / Admin / ClusterAdmin deltas are empty (no extra actions).

Templates: templates/user-authz-cluster-roles.yaml, templates/rbacv2/**.

Phase 2.2 — class policies (add-on)

Example embeddings-dedicated class with admission/scaling/update and authentication: None:

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: embeddings-dedicated
spec:
  admissionPolicy:
    allowedNamespaces: ["*"]
  exposurePolicy:
    type: ClusterLocal
    authentication: None
  scalingPolicy:
    allowedPriorityClassNames:
      - demo-inference-high
    minReplicas: 1
    maxReplicas: 4
  updatePolicy:
    strategy: RollingUpdate

An order under that class — it names no replica bounds, and the bounds of the class above apply to it:

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: search-embeddings
  namespace: ml
spec:
  inferenceServiceClassName: embeddings-dedicated
  model:
    ref: intfloat/multilingual-e5-large
    src: HuggingFace

Expected when Ready: status.constraints, an HPA with the replica bounds and a StatefulSet with its update strategy. status.endpoint.type mirrors exposurePolicy.type (ClusterLocal / External); API types such as Embeddings are out of scope for 2.2 (phase 2.3+ modelPolicy; later allowedEndpointTypes on the class, required and non-empty since phase 2.4.187; an order names no contract). F2-07 reasons: NamespaceNotAllowed. Scenario: internal-docs/ai-inference/scenarios/scenario-phase-2.2.md.

Phase 2.3 — ai-models integration (add-on)

Requires module ai-models with the internal F2-08 API when catalog integration is enabled. Public ModuleConfig uses catalog.mode (Enabled by default, or None to disable catalog clients). Internal transport defaults to aiInference.aiModels.catalogTransport: Rest: controller and planner call GET /api/internal/v1/models/lookup by model.ref. When Rest is unavailable, they immediately read Model / ClusterModel CRs in the same reconcile. Explicit rollback: internal catalogTransport: Kubernetes (CR-only). The distribution and catalog-import APIs are not used for model-facts lookup.

ModuleConfig (public catalog mode)

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: ai-inference
spec:
  settings:
    catalog:
      mode: Enabled                 # None — disable catalog clients / RBAC

Internal catalog transport (not ModuleConfig)

In openapi/values.yaml (module internal values, not public ModuleConfig):

aiInference:
  aiModels:
    catalogTransport: Rest         # default; Kubernetes — explicit rollback
    catalogLookup:                 # used when catalogTransport is Rest
      baseURL: https://ai-models-controller.d8-ai-models.svc.cluster.local:8080
      path: /api/internal/v1/models/lookup
      timeoutSeconds: 10
      notReadyRequeueSeconds: 30

InferenceServiceClass with modelPolicy

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: llm-chat-shared
spec:
  acceleratorPolicy:
    allowedDeviceClasses:
      - nvidia-h100-mps-mig
    allowedSharingModes:
      - Shared
    allowedPlacementTypes:
      - Partition
    maxAcceleratorCount: 1
  admissionPolicy:
    allowedNamespaces: ["*"]
  modelPolicy:
    allowedEndpointTypes:
      - Chat
    maxParameterCount: 70B
  exposurePolicy:
    type: External
    authentication: Token
  scalingPolicy:
    allowedPriorityClassNames:
      - demo-inference-normal
    minReplicas: 1
    maxReplicas: 1
  updatePolicy:
    strategy: RollingUpdate

InferenceService with catalog model.ref

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: support-llm
  namespace: support
spec:
  inferenceServiceClassName: llm-chat-shared
  model:
    ref:
      kind: ClusterModel
      name: qwen2-5-32b-instruct-awq
    src: ai-models

For a namespace Model, set kind: Model, name, and optional namespace in ref; the built workload does not carry the model namespace.

Controller behavior (2.3)

1. F2-07 2.1–2.2 (class and its policies).
2. Catalog fact lookup by model.ref (default — HTTP F2-08; when Rest is unavailable — immediate CR read; with Kubernetes — CR status only).
3. Local catalog F2-07 matrix (modelPolicy vs facts): on success ModelResolved=True; otherwise ModelResolved=False with reason (ModelNotFound, ModelNotReady, …).
4. The workload of the order is not built until ModelResolved=True.
5. On success the platform builds the objects of the order itself; the API contract it selected is published in status.model.endpointType and the scope of the model in status.constraints.modelScope.
6. API health probe and phase: Ready — same as 2.1–2.2.

Verify status

When Ready, expect:

• status.model.endpointType: Chat — the API contract the platform picked from the class allow-list
• status.constraints.modelScope: Cluster — from catalog facts
• status.endpoint.type: External — mirrors exposurePolicy.type, not the API type
• condition ModelResolved: True

kubectl get inferenceservice support-llm -n support -o jsonpath='{.status.model.endpointType}{"\n"}{.status.endpoint.type}{"\n"}'
kubectl get inferenceservice support-llm -n support -o jsonpath='{.status.constraints.modelScope}{"\n"}'

Scenario: internal-docs/ai-inference/scenarios/scenario-phase-2.3.md.

Phase 2.4 — resource planner (add-on)

Builds on 2.3 (catalog model.ref, ModelResolved). Adds F2-09 launch planning, condition Planned, and status.resolved. Requires module gpu (GPU inventory for F2-11) and the in-cluster resource planner Deployment shipped with ai-inference 2.4.

Compatibility: bundled default-llm and classes without acceleratorPolicy keep 2.3 behavior — the controller does not call the planner and does not publish Planned / status.resolved.

Prerequisites

• Ready ClusterModel / Model (2.3).
• Module gpu with PhysicalGPU inventory visible to the planner.
• Planner Service reachable at the module default (aiInference.planner.baseURL in chart values; see openapi/values.yaml).

InferenceServiceClass with acceleratorPolicy

Extend the 2.3 llm-chat-shared class with acceleratorPolicy (administrator manifest):

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceServiceClass
metadata:
  name: llm-chat-shared
spec:
  acceleratorPolicy:
    allowedDeviceClasses:
      - nvidia-h100-mps-mig
  admissionPolicy:
    allowedNamespaces: ["*"]
  modelPolicy:
    allowedEndpointTypes:
      - Chat
    maxParameterCount: 70B
  acceleratorPolicy:
    allowedSharingModes: [Shared]
    allowedPlacementTypes: [Partition, WholeDevice]
    maxAcceleratorCount: 1
  exposurePolicy:
    type: External
    authentication: Token
  scalingPolicy:
    allowedPriorityClassNames:
      - demo-inference-normal
    minReplicas: 1
    maxReplicas: 4
  updatePolicy:
    strategy: RollingUpdate

Expected: status.phase: Ready as in 2.3.

InferenceService with accelerator

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: support-llm
  namespace: support
spec:
  inferenceServiceClassName: llm-chat-shared
  model:
    ref:
      kind: ClusterModel
      name: qwen2-5-32b-instruct-awq
    src: ai-models

The branch of the compiled recipe is not an order field: the platform reads the latency branch for every order. The count of devices is not an order field either — the launch plan assigns it within acceleratorPolicy.maxAcceleratorCount of the class.

The share of a device is ASSIGNED by the launch plan, and the order names none. The plan takes the smallest whole percent that admits the memory the model needs on the chosen device, clamps it into minSharePercent/maxSharePercent of the class, and falls back to a whole device when the class permits no sharing, when the chosen placement is not a whole device in a sharing mode, or when either memory figure is unknown. The assigned value is published in status.resolved.sharePercent, so the owner of an order sees how much of the card is theirs. A share below 100 also requires the selected DRA device to publish allowMultipleAllocations with a sharePercent request policy. For a share of 1..99 the controller sends capacity.requests.memory from the model requirement of the launch plan, written into the claim as gpu.deckhouse.io/memory. Model memory and the MPS compute share are accounted independently.

Controller behavior (2.4)

1. F2-07 through 2.3 — class, catalog matrix → ModelResolved=True.
2. The accelerator policy of the class bounds the outcome of planning; the order has nothing of its own to check against it.
3. F2-09 POST /api/v1/launch-plan — planner returns placement + runtime recipe; on success:
   • status.resolved — the placement the order runs on: deviceClass, acceleratorProductName, acceleratorMemoryGiB, placementMode, sharingMode, sharePercent; everything else the planner computed travels in the launch plan of the order and is not published in the status;
   • condition Planned=True, reason LaunchPlanCalculated.
4. On planner refusal — Planned=False with stable reason (NoCapacity, QuantizationMismatch, …); the workload objects are not created or updated until planning succeeds.
5. After Planned=True — F2-10 applies the recipe from the plan into the arguments of the runtime container; then health probe → phase: Ready as in 2.1–2.3.
6. When maxReplicas > 1 — HPA observation and background preemption loop (roadmap p. 6) may adjust donor maxReplicas; bundled default-llm is out of scope.

Verify status

While reconciling, expect Planned before the workload of the order exists. When Ready:

status:
  phase: Ready
  resolved:
    deviceClass: nvidia-hopper-s3-shared
    acceleratorProductName: NVIDIA H100 80GB HBM3
    acceleratorMemoryGiB: 24
    placementMode: Partition
    sharingMode: Shared
    replanCount: 0
  conditions:
    - type: ModelResolved
      status: "True"
    - type: Planned
      status: "True"
      reason: LaunchPlanCalculated
    - type: Ready
      status: "True"

kubectl get inferenceservice support-llm -n support -o jsonpath='{.status.resolved}{"\n"}{.status.conditions[?(@.type=="Planned")]}{"\n"}'
kubectl get statefulset support-llm -n support -o jsonpath='{.spec.template.spec.containers[0].command}{"\n"}'

Planner refusal examples: Planned=False, reason: NoCapacity or QuantizationMismatch. Scenario: internal-docs/ai-inference/scenarios/scenario-phase-2.4.md.

Phase 2.4.1 — planner runtime fallback (add-on)

Builds on 2.4. When there is no entry in runtime-recipes.compiled.yaml for (family, model.name) and runtime fallback is enabled, the planner synthesizes a generic vLLM recipe from the images/catalogs/ tables (recipe-taxonomy.yaml and platform-hardware-families.yaml), estimates vramMinimumGiB, and returns recipeSource: fallback. Compiled presets and recipes always win over the fallback path.

Compatibility: with aiInference.planner.runtimeFallback.enabled: false (chart values / module internal config), behavior matches 2.4 — no synthetic recipe. Classes without acceleratorPolicy are unchanged.

Module settings

In openapi/values.yaml (not public ModuleConfig):

aiInference:
  planner:
    runtimeFallback:
      enabled: true   # default true

The controller forwards the flag in F2-09 as runtimeFallback.enabled. To disable:

aiInference:
  planner:
    runtimeFallback:
      enabled: false

Minimum metadata for fallback

The controller does not parse quantization from the name — only the planner on the fallback branch. For F2-07 modelPolicy.maxParameterCount, the controller judges whatever the path of the order resolved: the catalog fact on the ai-models path, the order value on the direct one, and the same model.name parsing when that first rung is empty.

Model and ClusterModel lookup facts do not return tensor geometry today. The planner gets geometry from a shipped recipe preset when one exists; otherwise the fallback estimate is geometry-blind. Publishing geometry from ai-models remains future work in #68125331; this module does not assume future catalog fields.

Example: model without a compiled recipe

ClusterModel named qwen2-5-32b-instruct-awq whose lookup facts omit parameterCount and quantization. The order also omits spec.model.parameterCount, so size and quantization come from model.name:

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: support-llm-fallback
  namespace: support
spec:
  inferenceServiceClassName: llm-chat-shared
  model:
    ref:
      kind: ClusterModel
      name: qwen2-5-32b-instruct-awq
    src: ai-models

Expected on successful F2-11 (GPU inventory populated):

status:
  resolved:
    deviceClass: nvidia-hopper-s3-shared
    acceleratorProductName: NVIDIA H100 80GB HBM3
    acceleratorMemoryGiB: 24
    placementMode: Partition
    sharingMode: Shared
  conditions:
    - type: Planned
      status: "True"
      reason: LaunchPlanCalculated

Without a recognizable size (no catalog, no spec.model.parameterCount, name without 32b/360m token) — Planned=False, reason NoCompatibleRuntimeAvailable.

Verify status

kubectl get inferenceservice support-llm-fallback -n support \
  -o jsonpath='{.status.resolved.deviceClass}{"\n"}{.status.resolved.acceleratorProductName}{"\n"}'
kubectl get statefulset support-llm-fallback -n support \
  -o jsonpath='{.spec.template.spec.containers[0].command}{"\n"}'

Cluster smoke: tools/smoke-phase-2.4.1.sh; fixtures under openspec/changes/phase-2-4-1/evidence/smoke-dev-ee/. See roadmap §2.4.1 and OpenSpec phase-2-4-1.