User Guide

This guide is for a namespace user: you order inference and read the result. It describes the module as it behaves today, in one pass. Which capability arrived in which phase is a separate subject — see Phase walkthrough.

What you do not do here: choose an accelerator, size a model volume, name a runtime or name an inference API contract. The platform assigns all of it. What you do is name a class and a model.

Quick start

1. Ask your cluster administrator which InferenceServiceClass your namespace may use. The class is cluster-scoped and you cannot read it — kubectl get inferenceserviceclass is not granted to a namespace user. The module ships a bundled class named default-llm.
2. Create an InferenceService in your namespace with a class reference and a model.
3. Wait for status.phase: Ready.
4. Read status.endpoint.url and, when the class requires a token, status.authSecretName.

apiVersion: ai.deckhouse.io/v1alpha1
kind: InferenceService
metadata:
  name: support-llm
  namespace: support
spec:
  inferenceServiceClassName: default-llm
  model:
    ref:
      name: Qwen/Qwen2.5-32B-Instruct-AWQ
    src: HuggingFace

An order is a class and a model, and nothing else

The spec of an InferenceService has exactly two required fields — inferenceServiceClassName and model — and no field for anything the platform decides. That is deliberate, not an omission:

• the inference API contract is picked from modelPolicy.allowedEndpointTypes of the class, intersected with what is known about the model. The chosen value appears in status.model.endpointType as Chat, Embeddings or Rerank. This is your only window on that choice, because the class is not yours to read. An order that needs a different contract needs a different class;
• the runtime and every launch parameter come from the recipe of the launch plan, compiled against the hardware the plan chose. A different value is a different recipe or a different class — an administrator’s decision;
• the accelerator, its share, and the cpu, memory and model-volume estimates are assigned by the platform and reported back in status.resolved.

An order that carries a runtime block anyway is rejected by a client asking for strict field validation, and silently pruned by one that does not.

Where the model comes from

spec.model.src selects the source, and the shape of spec.model.ref follows from it:

The API server validates this union before the object is stored, so a mismatch is refused at write time rather than surfacing later as a condition.

For a gated Hugging Face model, put the token in a Secret in your own namespace and name it:

spec:
  model:
    ref:
      name: meta-llama/Llama-3.1-8B
    src: HuggingFace
    authSecretRef:
      name: hf-token
      key: token   # optional; the default key is "token"

The controller validates that Secret while validating the order and mounts it into the model fetcher of the order workload as HF_TOKEN.

What happens after you create an order

1. The order is validated: the class must exist, be Ready and declare a non-empty allow-list of API contracts; the model reference must match its source.
2. The platform plans the order: it picks the accelerator, the placement and the share, and computes the cpu, memory and model-volume estimates. The result lands in status.resolved and in the Planned condition.
3. The controller applies the objects of the order workload in your namespace. Every object is named after the order and carries ownerReferences pointing at it, so deleting the order deletes them.
4. The endpoint is computed and published in status.endpoint.
5. When the class requires a token, a Secret named {order}-auth is created and named in status.authSecretName.
6. The Inference API is probed over HTTP. Only after a successful probe does the order reach status.phase: Ready.
7. After Ready, health is checked periodically. Sustained API errors move the order to Failed with reason ServiceUnhealthy; a failure before the first Ready keeps it at Pending with reason HealthCheckFailed.

Read the status

kubectl get inferenceservice support-llm -n support -o yaml

A ready order shows:

• status.phase: Ready
• status.endpoint.url — an HTTPS URL, with status.endpoint.type echoing the exposure the class chose (ClusterLocal or External)
• status.authSecretName — present when the class requires a token
• status.model.endpointType — the API contract the platform picked
• status.constraints — the effective replica bounds and priority class the class imposes on your order
• status.resolved — what the platform assigned: deviceClass, acceleratorProductName, acceleratorMemoryGiB, placementMode, sharingMode, sharePercent, and replanCount

The four phases mean:

Degraded is the one worth reading carefully: your endpoint answers. What does not work is growth.

When an order does not become ready

Read the conditions, not the phase — the phase says that something is wrong, the conditions say what:

kubectl get inferenceservice support-llm -n support \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.reason}{"\n"}{end}'

Five conditions answer in order: ClassResolved, ModelResolved, Planned, WorkloadReady, Ready. A sixth, ScalingHealthy, reports on growth rather than on readiness — True means healthy.

The reason is the answer. Common ones, and what each one asks of you:

• ClassNotFound, ClassNotReady — the class your order names does not exist, or it is not ready. Read spec.inferenceServiceClassName back and compare it with what your administrator ships; this is the reason to expect right after an upgrade that renamed the bundled class;
• ModelNotFound, ModelNotReady — the model reference does not resolve, or the model is not ready yet. Yours to fix;
• EndpointTypePolicyConflict — what the class allows and what the model offers do not intersect. You need a different class; the administrator owns the list;
• FormatNotAllowed, ParameterCountNotAllowed — the model is denied by the policy of the class. A different model, or a different class;
• CatalogDisabled — catalog integration is off in the module settings, and your order names a catalog model. Use a direct source, or ask for the setting to change;
• NoCapacity, NoAcceleratorWithEnoughMemory — the cluster has no room for this order right now. Waiting is a valid answer here;
• QuantizationMismatch, NoCompatibleRuntimeAvailable — no compiled recipe fits this model on this hardware. Not something you can resolve from the order;
• ResourceClaimExhausted — every device class the order was allowed to try has been tried and none holds it. Go looking for capacity;
• PlacementUndecidable — there was nothing to try from the start: the class cannot be judged at all. Waiting is not the answer — the selector expression of that class needs an administrator’s eye;
• ScaleUpBlocked — the order serves, but the autoscaler cannot grow it after preemption exhausted the capacity. This is the reason behind Degraded;
• LaunchPlanInForce — a reason of the Planned condition rather than of Ready, and an order carrying it is ready. It means the order runs on the launch plan it already holds while recomputing that plan did not succeed; what exactly did not succeed is in the message of the same condition. A serving order cannot be given its own place a second time — while it holds that place, the place is not published — so a refused recomputation speaks about capacity beyond it.

Every reason an order can carry, and not only the common ones, is answered on the FAQ page.

The distinction between the last two matters more than it looks. ResourceClaimExhausted means the search happened and came up empty — capacity is the subject. PlacementUndecidable means the search never started — the class is the subject.

Use the endpoint

URL=$(kubectl get inferenceservice support-llm -n support -o jsonpath='{.status.endpoint.url}')
SECRET=$(kubectl get inferenceservice support-llm -n support -o jsonpath='{.status.authSecretName}')
TOKEN=$(kubectl get secret "$SECRET" -n support -o jsonpath='{.data.token}' | base64 -d)

curl -sS "$URL/v1/models" -H "Authorization: Bearer $TOKEN"

When the class sets authentication: None, there is no Secret and no header.

Change or delete an order

The model and the class of an order can be changed by editing it; the platform replans and the controller re-applies the workload. status.resolved.replanCount counts how many times the plan has been recomputed.

Deleting the InferenceService deletes everything it owns — the workload, the service, the endpoint objects and the auth Secret — because they all carry ownerReferences pointing at it. The Secret holding your Hugging Face token is yours and is not touched.

Where to look next

• Examples — manifests to copy;
• FAQ — answers organised by refusal reason;
• Custom Resources — the full field reference.