Release notes

Product: ai-inference (Deckhouse Kubernetes Platform module) Stage: Experimental Date: 2026-08-05

Platform compatibility

The core resource.k8s.io/v1 DRA inventory is supported at that floor. Consumable-capacity fields remain optional: when the GPU driver does not publish them, the planner does not invent fractional capacity. This can reduce placement density, but prevents GPU overcommit.

This page describes the current experimental contour for operators and namespace users. It is the documentation canon for release notes. The Console feed is a short features / fixes projection of the same meaning.

Highlights

• Order an inference workload with cluster InferenceServiceClass and namespaced InferenceService; the controller deploys the objects of an order itself and publishes status.endpoint.
• An order is a class reference and a model. The inference API contract (Chat, Embeddings, Rerank and related) is picked by the platform from the allow-list of the class; the runtime configuration comes from the recipe of the launch plan.
• Plan placement and preemption through the planner API; autoscaling and donor restore stay under module ownership.

New features

• Bundled class and walkthrough path for a first Ready order.
• Placement preview, launch plan, preemption and free-replicas planner routes for operators and UI clients.
• Optional device-class and override surfaces on the order for GPU claim shaping.

Improvements

• Status conditions and readiness projection clarify order progress without exposing the platform’s internal churn as user contract.
• Shared estimate helpers keep host memory, CPU and PVC disk sizing consistent across controller and planner paths.
• The delivery image names the system packages whose files it carries, and carries their licences. Some files reached the image by copying, without the package records they belong to: the vulnerability scan did not see such files at all — not because it found them clean, but because it did not know they were there. The delivery now declares those packages by their records and carries the licence text of each at its canonical path, and no image leaves until the licence text of every declared package has been found.

Fixes

• The node resource estimate no longer charges the order for the weights of the model. The weights live on the accelerator and are already paid for as the VRAM requirement; on the node they are read as a stream through a page cache the kernel reclaims. Measured on an order of a hundred and twenty billion parameters: 66 GiB were assigned while the working set of the replica was 3663 MiB. The CPU estimate was read against the parameter count, which the work of the node barely follows — the accelerator computes, the node loads the weights and captures graphs — and eight cores were assigned against a steady draw of 0.083 of a core. The node now holds the runtime and the bounded window of its loader, and the CPU estimate follows the bytes read while loading. Above the small-model threshold no estimate yields a single core: the request receiver and the compute loop are separate processes, and on one core they take enough time from each other for the liveness probe to kill a healthy replica. The floor of vertical advice is read off the same estimate rather than a constant: advice may ask for MORE than the plan sized and never for less — it used to substitute its own, and an order received one core out of eight. Across a catalog of 363 models the sum of memory estimates fell from 17,644 to 2454 GiB with no model rising, and the largest estimate from 1268 to 14 GiB; fourteen models used to ask for more node memory than the largest node of the stand has.
• An order answering requests is no longer declared not ready because it is short of capacity to GROW. Recomputing the plan of such an order can be refused — when it needs more devices than it holds and none are free, say: the driver withdraws an occupied device from publication. That refusal used to put the order into Pending, end the reconcile before the objects of the order were built back, and erase the very condition by which the next pass decided whether to compute a plan at all — the order stayed not ready while it served, and never came out of it by itself. An order whose replicas hold their floor and their device now stays Ready: the Planned condition holds true with the reason LaunchPlanInForce, and the refusal, with its own words, sits in the message of that condition.
• Recomputing the plan of a serving order no longer trips over the order’s own place. The driver does not publish a device a claim has already selected, and the module used to drop the order’s own entry from the register of occupied capacity — the holder of the only device that fits stopped being visible to itself and was refused NoAcceleratorWithEnoughMemory while a replica served. The capacity an order holds is now handed to the planner apart and takes part in placing that order as free; a partition of an accelerator is recognized on a par with a whole card. The memory refusal also names the required amount and the largest one found, so «the model is too large for the fleet» and «the fleet shows nothing large right now» are told apart from one message.
• A device claim that could not be placed for a replica the order does not need no longer walks the order off the device class it runs on. Such a claim used to start a round of replans on behalf of the WHOLE order: the class landed in the register of excluded ones and the order was walked off its own place — and another place cannot be found for it without taking the set of replicas down, that is, without losing a working replica for one that does not exist yet. The subject of the round is now the claim the floor of the order needs; capacity missing for growth is announced as a GrowthClaimUnplaced event on the order and leaves it on its class. A claim whose device reports a failure does not count as placed, so a broken device under a replica of the floor starts the round as before.
• An order pinned to a MIG partition device class held by a single order is no longer refused with no way forward. Every partition profile in the hardware catalog pinned the shared sharing mode, while a partition device with no sharing attribute is identified as dedicated, so such a slot matched no profile at all and placement answered NoCompatiblePlacement on healthy hardware with memory to spare. Partition profiles no longer pin a sharing mode: one description now serves both kinds of slot, and the memory reserve for sharing a partition’s compute is subtracted from the slot’s actual mode — the shared slot still pays it, the dedicated one does not.
• Placement no longer answers differently for the same cluster from one snapshot to the next when a partition is published both as a dedicated and as a shared device class. Such a pair used to be able to tie in the candidate comparison, and a tie was resolved by whichever slot the inventory happened to return first.
• An order asking for a context horizon above the model’s window is now refused by name instead of diverging. runtime.<endpointType>.maxModelLen was checked only against the ai-models catalog fact, so on the direct Hugging Face path nothing bounded it. The planner then sized the VRAM budget for the clamped horizon while the pod was launched with the requested one, and the runtime failed to start with no reason pointing at the order. The check now runs for both model sources and against the narrower of the two windows the platform knows — the catalog fact and the recipe preset — and the refusal message names which of them bound the decision.
• An uncalibrated inference runtime no longer receives an invented VRAM overhead. An unrecognized runtime name silently got 0.5 GiB, which under-sizes the budget; it is now a named refusal. No shipped asset can reach this path today, so the change is defensive.
• The minimal order now works on the bundled class. It declared no modelPolicy, so an order that named only a class and a model was refused with EndpointTypeAmbiguous — a reason that is not retried, leaving the order Failed until edited by hand. The bundled class now declares modelPolicy.allowedEndpointTypes: [Chat], so the single allowed type is selected automatically and the two-reference order reaches Ready.
• A model size stated on the order can no longer understate the catalog. The effective size used to be min(order, catalog), so an order could only ever lower it — which both slipped past modelPolicy.maxParameterCount and undersized the memory budget handed to the planner. When the catalog holds a fact about the model, that fact is now the effective size; the order value applies only where no fact exists.
• Pod memory and CPU come from one estimate — the one the planner computed and put into the launch plan of the order. The bridge used to compute them separately and missed the CPU KV cache headroom a recipe asks for: for google-gemma-4-* models on intel-cpu the memory request grows by the 40 GiB the recipe declares, and the vpa.resourcePolicy.maxAllowed ceiling doubles that. Pinning resources on the order still wins over both sources.
• Recipe environment variables are no longer dropped wholesale. A boolean recipe key was written into status as true/false while the reader expected 1/0, and the failure was replaced with an empty value — the pod lost its entire env, including variables set on the order. A value the catalog cannot read now refuses with reason RuntimeParameterNotSupported instead of launching without the variables.
• Preset context horizons now match the model window. For microsoft-phi-3-5-mini-instruct the window itself was understated — declared 4096 against a real 131072 — so the pod served 4096 instead of 8192; the window is curated and the horizon now reaches the pod in full. For microsoft-phi-3-mini-4k-instruct and microsoft-phi-3-medium-4k-instruct the 4096 window is correct and the horizon was lowered to it; behaviour of those two models does not change.

Upgrade notes

• The platform no longer keeps a list of accelerator names. The kind of hardware is derived from what the device says about itself — the generation from the resource snapshot, else a prefix of the product name, else a refusal. Every device of a declared family becomes orderable, not only one whose name somebody had written down. One thing is required of you, and only if you read the profileId field of a launch plan or of a placement preview: the name of a derived record spells the full published name of the device where the withdrawn record spelled a shortened marketing one. Measured on the withdrawn list: of forty-seven partition records the spelling moved for eighteen — a100-40gb became a100-sxm4-40gb, gb200 became gb200-nvl4. The order status carries no such field, and nothing else observable changed.

• An order placed before the upgrade on an older platform re-plans once. Such an order keeps in its status the pattern of the record it was placed by rather than the name of a device; a pattern no longer resolves, the order re-plans and its status is rewritten with the name of the card. No manual step is needed.

• The cluster view names the hardware kind of every accelerator. The accelerator identity gained identity.hardwareKey — the kind the platform assigns to this card. It is empty when the rules lead nowhere: such a card cannot be ordered, and the view says so rather than staying silent. The resolution of a kind also began to judge a fact rather than a spelling: before this, a card naming a generation that happens to spell out a key reached a kind it never named. Nothing is required of you: the identity fields you already read are unchanged, and a reader that does not know the new field behaves as before.

• An order on hardware the wheel carries no kernels for is refused before it starts. A device names its own compute capability, the family register declares the capabilities the kernels are compiled for, and the platform compares them. Such an order used to be accepted and to die at start: the binary of one capability does not run on a device of another. One thing is required of you: an order refused NoCompatiblePlacement on hardware that used to be accepted does not need a retry but support for that capability in the delivery, which is declared by the source contract of the runtime. A limit is worth knowing too: a device whose capability the snapshot does not name is judged as before — by its kind of hardware alone — so an installation with an older resource driver does not change behaviour.

• The platform now recognizes hardware by what the hardware says about itself. A device whose generation the resource snapshot names gets its kind of hardware without an entry in the register of names. Hardware that used to be refused becomes orderable: measured, NVIDIA H20, NVIDIA H800, NVIDIA A16, NVIDIA A2, NVIDIA L2, NVIDIA L20, NVIDIA GeForce RTX 4090, NVIDIA RTX 6000 Ada Generation and AMD Instinct MI300X matched no entry of the register at all, although their families were declared supported. Entries of the register still win over the derivation, so not a single kind recognized before has changed. One thing is required of you: an Intel accelerator the platform does not recognize now gets a NoCompatiblePlacement refusal rather than the kind of a central processor — the old substitution handed an order a recipe compiled for other hardware. A limit is worth knowing too: the platform does not know what share of the devices in the field publish a generation, so the kind is derived by a ladder — the generation, else the product-name rule, else a refusal — and hardware no step of the ladder reaches is refused before it starts.

• The accelerator name in the status of an order is now the name of the card. The field status.resolved.acceleratorProductName used to carry the pattern of a hardware record rather than the name of a device: an order placed on NVIDIA H100 PCIe reported NVIDIA H100*. It now carries what the device calls itself, and there is no star in the value any more. The same holds for status.resolved.acceleratorVendor and for the name in the placement preview. One thing is required of you: if you have a watcher comparing this value against a string with a star, rewrite the comparison against the name of the card. A second thing is worth knowing: donor selection for preemption compares this name, so a target and a donor on different cards of one kind (say NVIDIA H100 PCIe and NVIDIA H100 80GB HBM3) are no longer treated as compatible — they used to be, and the target was offered a donor whose freed card it cannot use. An order planned before this release is not offered as a donor until its placement is computed again; that happens on a change of the device inventory or an edit of the order.

• Orders will ask the node for different resources. The estimate is recomputed on the first pass after the upgrade, and the new figures reach a replica when that replica is next replaced — a scale-out, an edit of the order, a lost node. Memory moves one way only: 166 models of the catalog fall, 197 are unchanged, none rise. Cores move both ways: 95 models fall and 43 gain one core, always the step from one to two. Nothing is required of you, but a node packed to its CPU limit with orders from those forty-three will hold fewer of them; the sum of core estimates across the catalog falls from 864 to 639.

• The Planned condition is now true in two ways. Besides the familiar LaunchPlanCalculated — a plan computed by this pass — there is LaunchPlanInForce: the order is assembled from the plan it already holds, and recomputing that plan was refused. Nothing is required of you. One thing is worth knowing: a watcher that picked out orders with no place by Planned=False will no longer find a serving order whose recomputation was refused — such an order is now Planned=True, and the refusal sits in the message of the condition and in the counter of validation failures.

• A new runtime release no longer restarts the replicas that are serving. An upgrade that brings a new runtime release carries it into the description of the workload on the first reconcile, and the replicas already running are left on the release they were started with. A replica raised after that — a widened set, a replacement after a node went away — comes up on the shipped release. While the two differ the order carries RuntimeCurrent=False, reason ReplicasHeldOnPreviousRuntime, and the message names both releases and how many replicas reached the shipped one. Nothing is required of you; editing the order moves every replica the way its class asks, and deleting a replica moves that one. Two things are worth knowing: while the set is mixed the replicas answer differently near the context length and the concurrency limit, because those come from the recipe of each replica’s own release; and if the recomputed plan moves the placement of the order — another device class, another share, a larger model volume — the replicas cannot be kept at all, the order reports reason WorkloadRebuilding and goes through Pending. Before this release the shipped runtime never reached a steady Ready order at all: it arrived at whatever later moment something else touched the order, which is to say the restart happened outside any upgrade window. See ADMIN_GUIDE, «Upgrading the module».

• Breaking. updatePolicy.strategy: Recreate is no longer accepted. The value named nothing the workload object of an order knows — a StatefulSet takes RollingUpdate or OnDelete — so a class declaring it had every apply of every workload of its orders refused, and the refusal reached the owner of the order rather than the author of the class. The schema now accepts RollingUpdate only, and a class stored earlier with Recreate is refused by validation with the strategy named in the reason. Fix such a class before upgrading; orders of classes naming RollingUpdate, or no update policy at all, are untouched.

• Breaking. The bundled class is renamed, and an order naming the old name loses its class. The bundled InferenceServiceClass is now default-llm; the old name was default-llm-chat. Helm creates the object under the new name and removes the old one, so an order whose spec.inferenceServiceClassName still names the old one reports that its class does not exist. Replace the name in your orders before upgrading: nothing migrates the references for you, and there is no grace period in which both names answer. What you gain by it. The same class now permits every inference API contract the module supports — chat, embeddings and reranking — instead of chat alone. An order names no contract; the platform picks it from the list of the class, narrowed by what the source of the model says the model can serve. So on the catalog path an embedding model is served as an embedding model, with no class of your own. What still needs a class of your own. On a source that carries no facts about the model, nothing narrows the list and the FIRST contract of it wins — chat. An order of an embedding model on such a source is served as chat, silently. If that is your case, declare a class whose list names the one contract you need.

• Breaking, and the transition is dangerous by silence. Two fields leave the spec of an order: spec.launchStrategy and spec.priorityClassName. The schema of an order is structural and keeps no unknown fields, and the admission policy on write checks only the length of a name — so an order still naming those fields is accepted while the fields vanish from the stored object without a word. There will be no error by which an owner would learn about the transition: fix your manifests before upgrading. Where to get the answer. The branch of the compiled recipe cannot be set anywhere: the platform reads the latency branch for every order. The priority of a workload comes from the class allowedPriorityClassNames allow-list, and the chosen name is visible in status.constraints.priorityClassName.

• Breaking. Three sub-blocks leave the resources of an order: spec.resources.requests, spec.resources.limits and spec.resources.storage. The platform ESTIMATES the CPU and memory of the inference container and the floor of the model volume from the size and the quantization of the model, and a value named on the order used to cancel that estimate whole — one key of cpu or memory switched the whole overlay off. The applied numbers are visible on the workload of the order. The size of the volume is the sharpest of the three: a volume cannot be resized afterwards, so an order sized by hand stayed wrong for ever.

• The StorageClass of the model volume moves to the class. Nobody computes where a model should lie, so the decision belongs to an administrator: spec.modelStorageClassName on InferenceServiceClass. A class that names none leaves the volume to the default StorageClass of the cluster, and the platform adds no second default.

• A volume that already stands keeps its size, as long as that size still serves the order. The size is settled when the volume is CREATED and cannot be changed afterwards — the server refuses the whole apply of a workload naming another size. So the platform reads the standing workload and carries its claim templates forward instead of computing them again. Which way the divergence goes decides what happens:

  • the volume is larger than the estimate — it is kept and the order stays ready. Nothing is asked of you, no claim is resized or deleted, and the order gets a SettledRegionKept event naming both sizes. The claim templates travel whole, so a modelStorageClassName the class has since changed is kept as well: a volume does not move to another StorageClass, and a standing order keeps the one it was made on;
  • the order has outgrown the volume — the model no longer fits, so nothing is carried and the order is refused rather than reported ready on a volume it cannot use. The order gets a SettledRegionTooSmall warning naming both sizes; the server’s own refusal only says that the field is forbidden. To get the order moving, delete BOTH its workload and the claim behind it — for the order chat that is the StatefulSet chat and the PersistentVolumeClaim storage-chat-0. The next pass builds the workload back at the size the order now needs, and the artefact is downloaded again. Deleting the workload alone is not enough and leaves the order worse off: the claim outlives it, the rebuilt workload binds to that same claim, and the order then runs on the old volume with no warning at all — the two claim templates now agree, so there is nothing left to report.

• How the removals above are refused, measured rather than assumed. A client that asks for strict field validation — which kubectl apply does by default — gets a manifest naming any removed field REFUSED, and the message names the field. A client that does not ask for it gets the object accepted with the field pruned away in silence. Both are true; which one you meet depends on your client, and automation on a client library is the half that gets the silence.

• Breaking, with a DELAYED consequence. The block of replica bounds leaves the spec of an order: spec.scaling. As with the fields above, an order still naming it is accepted and the block vanishes from the stored object without a word — but here the effect is not immediate. An order working in three replicas keeps them until the bounds are recomputed, and only then drops to what its class allows. The moment of that recompute is not something you can read off the object, so fix your manifests rather than watch for it. Where to get the answer. The bounds come from spec.scalingPolicy of the class, and the effective values are visible in status.constraints.minReplicas and status.constraints.maxReplicas of your own order. An order that needs its own bounds needs its own class, and that is an administrator’s decision.

• A class without a scaling policy gives one replica. This is declared behaviour now, not the leftover of an unset field: an order names no bounds, and the platform has no second source. For administrators: a class under which orders have to work in more than one replica must declare spec.scalingPolicy. The bundled class receives it in this release (one to two replicas).

• The refusal ScalingOutOfBounds is gone, because nothing of the order is left to compare with the policy of its class. A class whose own policy crosses its bounds is still refused — as the fault of the class, under InvalidServiceClass.

• Breaking, with a DELAYED consequence, and one capability moves from the order to the class. The accelerator sub-block leaves the spec of an order, and the shell of the resources block goes with it: spec.resources.accelerator.count, spec.resources.accelerator.sharePercent, spec.resources.accelerator.deviceClasses and spec.resources itself. A strict client is refused naming spec.resources; a lenient one is accepted with the block dropped in silence. The consequence is delayed: an order running on two devices keeps them until the next planning, and only then gets what the plan assigns. Where to get the answer. The launch plan assigns the count of devices and the share of one for the chosen model on the chosen hardware, inside the limits of spec.acceleratorPolicy of the class. The assigned share is visible in status.resolved.sharePercent of your own order, and the assigned device class in status.resolved.deviceClass. The allow-list of device classes is the one of the class, and an order that needs its own needs its own class.

• Two refusals are gone, because neither has a subject any more: AcceleratorCountNotAllowed and AcceleratorShareNotAllowed. The limits of the class did not change and did not weaken — they now bound what planning assigns instead of what an order asked for. A bound that does not admit the memory a model needs gives a whole device rather than a refusal.

• Orders that stood refused will START WORKING. An order asking for more devices, or for a share outside the bounds of its class, was refused and stayed refused. There is nothing left to ask with, so such an order now goes through and gets what the plan assigns. If you kept one of those refusals as a guard against an order you did not want running, the guard is gone: use the policies of a class of its own instead.

• A class without an accelerator policy gives a whole card. Splitting a card is permitted by the class and by nothing else now: an order asks for no share. For administrators: a class whose orders should share accelerators must declare spec.acceleratorPolicy allowing Shared together with WholeDevice. The bundled class receives such a policy in this release.

• The result: an order is a class reference and a model. The runtime block leaves the spec of an order whole — the API contract and the name of the runtime were its last two fields. A strict client is refused naming spec.runtime; a lenient one is accepted with the block dropped in silence. Where to get the answer. The API contract is picked by the platform out of the allow-list of the class intersected with the facts of the model: the only allowed contract, or the first of several. The chosen value shows up in status.model.endpointType — the one window the owner of an order has on that choice, because the class is not theirs to read. An order that needs a different contract needs a different class. The name of the runtime is picked by the recipe.

• A second breaking change, and it is on the other surface: a class must declare a non-empty allow-list of API contracts. A class without one is refused on write, and the message names the field. It is a refusal addressed to the administrator rather than to the owner of an order: the contract is picked from that list, an order names none, and a class without it would leave its orders with no contract at all. Hand-written classes without the list will stop being accepted on their next edit. Declare spec.modelPolicy.allowedEndpointTypes with at least one contract. The bundled class declares it, and so does every class shown in the examples, so the edit only touches hand-written ones.

• The refusal about an ambiguous API contract is gone together with the order field that used to rescue it. An order under a class with no list used to be refused in a way its owner could not fix. Such a class is now refused on write, and a class stored before the requirement refuses the order as a fault of the class.

• Configuring the runtime from an order is no longer possible — the capability is withdrawn in full. Three parameter sections leave the spec of an order: spec.runtime.chat, spec.runtime.embeddings and spec.runtime.rerank, with every field they carried — the context length, the number of concurrent sequences, the KV cache dtype, the output token limit, the temperature, the nucleus fraction, trusting remote model code and the pooling kind. A strict client is refused naming the section; a lenient one is accepted with the section dropped in silence. Where to get the answer. All of these values are set by the recipe of the launch plan, compiled against the hardware that was chosen: the context length and the sequence count depend on the memory of the card, and the cache dtype on what the hardware supports. A different value is a different recipe or a different class — an administrator’s decision.

• Breaking, and this removal GIVES a value back rather than taking one away. The process environment leaves the spec of an order: spec.runtime.env, with its two keys cpuKvcacheSpaceEnv and logLevelEnv. A strict client is refused naming spec.runtime.env; a lenient one is accepted with the object dropped in silence. Every environment variable of the inference process now comes from the recipe of the launch plan, and the order of precedence is two rungs instead of three: the recipe, then the internal defaults of the module. Where to get the answer. A value that has to be different is set by the recipe. An order with its own requirements for the environment needs its own recipe or its own class, and that is an administrator’s decision.

• The key you REMOVED comes back. This is the sharp edge of both removals — the parameter sections and the environment object — and it has no analogue among the ones above. An order could drop a key inherited from the recipe by writing an explicit null into it; with the section and the object gone there is nothing to drop it with, so the recipe value returns. The key does not disappear from the pod — it APPEARS in it. If you were suppressing a recipe value that way, the pod will run with it on the next assembly of the workload.

• The refusals LegacyExtraEnv and LegacyExtraArgs are gone. Each caught a manifest still carrying a legacy array — of environment variables or of launch arguments — and pointed at a migration target: the closed environment object and the parameter section. Neither exists any more, so the refusals could only point at nothing. Such a manifest is now refused by the schema itself, as an undeclared field.

• The refusal RuntimeParameterNotSupported stays, but its subject is a different one now. It no longer concerns an order: there is nothing of an order left to check. It names a key of the RECIPE that the parameter catalog cannot read — and the order is refused rather than launching a pod on the runtime’s own defaults.

• A narrowed capability. A throughput order can no longer be placed: the Throughput and Balance branches are not selectable by an order. Compiled recipe entries still carry those branches and the branch key of the launch-plan request is unchanged — an order simply no longer sets it.

• Priority now differs between classes, not within one. Two orders of one class used to differ in priority through the order’s own field; the priority now comes from the class allow-list, so two orders of one class are equal in priority and neither can be a donor for the other. Different priorities are expressed by different classes.

• Breaking: a refusal on write for catalog orders. The size of a model (spec.model.parameterCount) and the credentials reference (spec.model.authSecretRef) are accepted only when the source is HuggingFace. An order through the catalog carrying either is refused on write, and the message names exactly the field that does not belong. The size comes from the facts of the catalog entry; the credentials for fetching a catalog artifact live with the catalog itself. On the direct path both fields work as before.

• Breaking. The composition of status.resolved on an order is now closed: it keeps the placement the order runs on — deviceClass, acceleratorProductName, acceleratorMemoryGiB, placementMode, sharingMode — and the state of the planning episode: replanCount, excludedDeviceClasses. Thirteen fields are gone: profileId, hardwareKey, quantization, acceleratorVendor, vramRequiredGiB, acceleratorCount, launchStrategy, recipeSource, recipeConfirmed, modelStorageGiB, hostMemoryGiB, hostCPUCores, deviceClassesNotAllowed, and with them the nested runtime block. The whole status.appliedClass block is gone as well. An expression addressing a removed path returns an empty value rather than an error, so check your queries before upgrading. Where to get the answer if you needed one: the name of the applied class is in spec.inferenceServiceClassName of the same order, where you set it; the scaling policy of the class is on the class, for those allowed to read it; the effective replica bounds, the priority and the model scope stay on their paths in status.constraints and status.model. The remaining removed values are internal: the planner computes them and the reconcile of the order takes them from the launch plan. They had no external reader, and the module will not publish them again — the surface of an order answers “where did my order land and am I alone on the hardware”, it does not restate the answer of the planner.

• Breaking. The spec.settings.maas settings block leaves the module’s values schema, together with its packageRepositoryName and packageVersion keys. A key the schema no longer declares is rejected by the API server: a ModuleConfig that sets it will not apply after the upgrade. Remove the maas block from the module settings before upgrading. There is nothing to put in its place and nothing is needed: what the repository and version were pinned for, the module now does itself — it builds the images of the request runtime and of the model artifact fetch and names them by digest, and it deploys the objects of an order without an intermediary.

• An order that names the device class maas now gets exactly that class. The name used to read as «no device class named» — it was the default value of the foreign chart — and the order was resolved as if it had asked for none. If you have a device class of that name and an order for it, the order will land on it after the upgrade.

• Breaking. The readiness condition of an order is renamed from ApplicationReady to WorkloadReady, and its two failure reasons from ApplicationNotReady and ApplicationError to WorkloadNotReady and WorkloadError. Replace the three names before upgrading; the same applies to queries over ai_inference_validation_failures_total that select the old reason names. Nothing has to be re-created: an order already in the cluster carries the new condition after its first reconcile on the new version, and the same reconcile removes the entry of the old condition type from status.conditions. Until it runs, that entry is still there holding the value it last had — True for an order that was working — so automation must not read it while waiting for the upgrade to roll out.

• Breaking. The scaling condition of an order is renamed from ScalingLimited to ScalingHealthy, and its polarity is reversed: True now means healthy scaling, False means a scaling limit. The old name is not gone from the contract — it became the REASON on the negative side, beside Preempted, and the condition of the Kubernetes HPA keeps its own name and its own meaning. Renaming alone is not enough: every check of the status has to be turned over too. An expression that looked for True as the sign of trouble must now look for False, and one left as it was returns the healthy orders instead of the limited ones — with no error to show for it. Nothing has to be re-created: an order already in the cluster carries the new condition after its first reconcile on the new version, and the same reconcile removes the entry of the old condition type from status.conditions. Until it runs, that entry is still there holding the value it last had — False for an order that was working, which reads as a refusal under the new polarity — so automation must not read it while waiting for the upgrade to roll out.

• The module now deploys the objects of an order itself instead of asking a delivery package to do it, and their names lose the -app suffix: the order chat is served by the workload chat, not chat-app. This is not a field of the contract, but it is visible to anything that reached the objects of an order by name — dashboards, metric queries, hand-written kubectl. The app label on the pods carries the same new name.

• The workload of the old path stays in the cluster, and removing it is yours. The module no longer holds the permission to touch it and no longer knows its name: it deploys and prunes only the objects it builds itself, and an order is not re-created by an upgrade, so nothing garbage-collects the old set either. The old {order}-app workload therefore keeps running beside the new one, and its pods keep holding the device claim they were given. On a limited device pool that matters twice over: the devices are consumed by two workloads instead of one, and the new claim of an order may find nothing left to allocate — an order that was Ready before the upgrade can stay Pending after it. Remove the leftover set per order after the upgrade, checking first that the new workload serves the order. Go by the object NAME, not by a label: the old set and the new one carry the same app label — the name of the order — so a label selector cannot tell them apart. The old objects are the ones whose name ends in -app:

  kubectl -n <namespace> get inferenceservice <order> -o jsonpath='{.status.phase}'
  kubectl -n <namespace> get statefulset,service,ingress,poddisruptionbudget,secret,\
    resourceclaimtemplate,servicemonitor,horizontalpodautoscaler,verticalpodautoscaler,certificate \
    -o name | grep -- '-app$'
  kubectl -n <namespace> delete application <order> --ignore-not-found

  Deleting the Application object is not enough on its own — the rendered objects carry no owner reference back to it — so remove what the second command listed as well, once you have read the list.

• spec.model.parameterCount and InferenceServiceClass.spec.modelPolicy.maxParameterCount now require an explicit M or B suffix (case-insensitive). Manifests that use a bare value such as "32" are rejected by the API server after the upgrade. Replace each bare value with the intended scale, for example 32B or 350M, before upgrading the module.

• Preemption now reads the scheduler’s own verdict when it counts pods blocked on placement: a pod counts as blocked when it carries PodScheduled=False with reason Unschedulable. Before this release the count was inferred from “phase is Pending and no node assigned”, which also counted a pod the scheduler had not looked at yet — every pod looks like that for the moment between creation and the first scheduling attempt. The donor-selection rule itself did not change: it still keys on DRA claim state, so no order is preempted that was not preempted before. What changes is the number reported in preemption events, which is now the count of pods the scheduler actually refused. No metric carries that number.

• Capacity preemption is now observable. The module emits an event on both objects — on the donor whose replica bound was lowered, naming the recipient, and on the recipient, naming the donor — and the donor’s ScalingHealthy condition message names the recipient as well. Both events are published only when capacity actually moved: a pass the planner approved that released nothing is counted as its own outcome (result="noop") and stays silent, so an event no longer announces a preemption that did not happen. When the hold ends, the donor gets a closing event naming the bound it was restored to and saying whether the restore was full or partial; a hold that cannot be undone, because the child object is gone, gets its own event instead of falling silent. A partial restore keeps the ScalingHealthy=False condition with reason Preempted, since the donor is still held. Two alerts ship for the first time: capacity was taken, and a donor is held at a lowered bound; the new gauge behind the second one is ai_inference_preemption_donors_held. A refusal reason outside the declared allowlist now lands in an Other bucket on ai_inference_preemption_failures_total instead of being dropped; before this release it was silently discarded while the attempt counter still recorded the refusal, so the two never added up. ai_inference_preemption_attempts_total{result} now carries six values — ok, noop, refused, error, restored, hold-cleared — of which two are new: noop for a pass that moved nothing, and hold-cleared for a hold that ended because the child object is gone. Both used to be counted as something else, ok and error respectively, so a query on either of those two counts fewer events after the upgrade. The counter also creates every value of a device class at zero as soon as the class is first observed, so count(), absent() and sum by (result) see series for outcomes that have not happened, and a device class the planner named on a pass that preempted nothing still appears. It does NOT let increase() see the very first preemption on a device class this controller process has never seen before: that series appears and is incremented between two scrapes, and no in-process initialisation can put a zero sample between them. An earlier edition of this note claimed otherwise. Preemption metrics gain a deviceClass label. The result label is not renamed, so a query naming it still selects the right series — but its VALUE set grew, so a query on result="ok" or result="error" counts fewer events than before, as the paragraph above spells out. Object names are deliberately not metric labels — identity lives in the events and in the condition, where it costs no metric cardinality. The device-class label is spelled deviceClass, camelCase, the same as the field of the resource definition. The events are named CapacityPreempted (donor), CapacityGranted (recipient), CapacityRestored (donor, hold over) and PreemptionHoldCleared (donor, hold ended with no restore); the alerts are D8AIInferenceCapacityPreempted and D8AIInferenceDonorHeldAtLoweredBound. The runbooks name CapacityPreempted, CapacityRestored and PreemptionHoldCleared in their kubectl commands; CapacityGranted is read with describe or in the events feed.

• The HPA observer no longer writes the scaling condition on an order held by preemption. Two writers of one condition meant the reason Preempted — which the hold alert’s runbook queries by — was replaced with ScalingHealthy on exactly the donors where it matters. Visible in two places: a held donor’s condition now keeps the preemption reason and its message, and ai_inference_hpa_observation_total counts such a pass as skipped.

• The set of events that wakes the preemption loop changed in one half: it now wakes on a change of the BLOCKED verdict — PodScheduled=False with reason Unschedulable — where the same half used to read “phase is Pending and no node is assigned”. The verdict counts in BOTH directions: entering the refusal wakes the loop and so does leaving it. What does not wake it through this half is a transition that leaves the verdict unchanged — between two states neither of which is a placement refusal. Waking on a pod phase change is unchanged, and so is waking on pod creation, which is unconditional and always was. Nothing has to be reconfigured, and no first reaction is delayed: what changes is WHICH updates wake the loop through this half. A pod the scheduler has refused now produces one wake-up at the moment the verdict arrives, which the old rule did not (it already considered such a pod blocked from creation); a pod that goes from unscheduled to scheduled without ever being refused produces none, where the old rule woke the loop when the node was assigned.

• History of that release. The delivery bridge pruned settings keys of its child object that it did not want, and the effect was visible on an upgraded cluster. A key an administrator had added with kubectl edit (an Update, not an Apply) was removed on the next reconcile, because such a key belonged to a manager named after the process and was indistinguishable from a leftover of the module’s own. The bridge deferred the autoscaling key to another apply manager only while a preemption episode was actually running, and sent it as usual once the episode was over. A key written with a server-side apply under its own field manager was left alone. The bridge itself has since been removed — the platform builds the objects of an order itself — so nothing of this paragraph describes the module today. The advice it gave still holds for the objects the platform builds: keep manual settings under your own field manager (kubectl apply --server-side --field-manager=<your-name>) rather than editing them in place.

• An order whose allocated GPU reports a failed device now triggers a replan. The controller adds the current device class to the exclusion list, asks the planner again, then deletes and rebuilds the workload of the order with the new device — so it restarts on a different card instead of staying on a broken one. Before this release the failure was only acted on when the claim had no allocation, a combination Kubernetes 1.35 and later cannot hold: the API server refuses a device condition for a device that is not allocated in the claim. Orders whose claim never got allocated are unaffected — those were, and still are, caught by the pending grace window. An allocated device that merely reports Ready=False does not trigger a replan: that is how a freshly allocated device looks while the driver configures it, and replanning there would move a healthy card off its device class for good. Only Failed=True on an allocated device is treated as a failure.

• The context-horizon check now also applies on the direct Hugging Face path. An existing order whose maxModelLen exceeds the window declared by the model’s recipe preset will move its order condition to False with reason RuntimeParameterNotSupported at the next reconcile that bumps the generation. Note the asymmetry: only status is patched, so the workload of the order and its pod stay in place — the order reports a failure while the workload keeps running. Lower maxModelLen to the model window, or drop the key so the launch strategy default applies. Four models shipped in the recipe presets declare a window today, all in the microsoft family; orders on any other model are unaffected.

• The bundled class now restricts the API endpoint type to Chat. Before this release the class declared no restriction, so an order for a catalog model whose types did not include Chat was silently resolved to the first type the catalog offered. Such an order is now refused by name. The class is meant for generative chat, and the declared set is brought in line with the name; for another endpoint type declare your own InferenceServiceClass. An existing Ready order keeps running until its next change bumps the generation. Superseded. The bundled class now permits every contract the module supports, and its name no longer states one of them. What that changes for the choice of a class is in the upgrade note at the top of this page.

• On the bundled class, an order that does not state spec.runtime.endpointType may now resolve to a different API endpoint type than before, with no refusal and no warning. Before this release the type was the first entry of the model’s catalog supportedEndpointTypes; now it is the intersection with [Chat]. For a model advertising [Embeddings, Chat] the resolved type changes from Embeddings to Chat, which changes status.model.endpointType, the recipe branch and the runtime parameters. State spec.runtime.endpointType explicitly, or use your own class, if you depend on a specific type. Note the previous behaviour depended on the ordering of types in the catalog rather than on anything in the order.

• An order that states spec.model.parameterCount below the catalog fact will now be sized against the fact. Requested host memory, CPU, model storage and video memory grow accordingly, and a class maxParameterCount limit that the understated value slipped past will now refuse the order.

• Breaking for QUERIES over measurements, and silent. Two values of the result label on ai_inference_reconcile_total change spelling: application_ensure_transient becomes workload_ensure_transient, and application_error becomes workload_error. They named the delivery package removed whole in an earlier slice, while the workload stages beside them already named the workload. Nothing in the module reads this counter, so nothing inside it breaks — a query of YOUR own that selects an order by the old spelling will find zero events, not an error, which is the dangerous half: a silent query looks like «nothing happened». Rewrite such a query to the new spelling before upgrading. The series of the old value does not disappear from your measurement store; it stops growing, so a query over a historical window stays good. Nothing is recreated, and no order changes because of this.

• The module is Experimental: expect additive CRD and settings changes between phase slices. Prefer the walkthrough and OpenAPI/CR reference for the current field set before upgrading a live order.

Documentation

• Overview: Overview
• Phase history: Phase walkthrough
• Custom resources: Custom Resources
• User guide: Usage
• Administration guide: Administration
• Examples: Examples
• FAQ: FAQ