Administration Guide

This guide covers administrative tasks: enabling the module, connecting S3-compatible storage, choosing model delivery, preparing node-local cache, configuring distribution, and checking runtime health.

Requirements

Minimum Deckhouse Kubernetes Platform and Kubernetes versions are listed under Requirements on the Configuration page.

• S3-compatible object storage and a bucket for DMCR data and model preparation staging data. One bucket serves one cluster (see Object storage and DMCR).
• Secret in d8-system with accessKey and secretKey.
• RWX StorageClass with volumeBindingMode: Immediate for SharedPVC (see Storage prerequisites).
• sds-node-configurator and sds-local-volume modules for NodeCache.

Enablement

Create a Secret with object storage credentials:

apiVersion: v1
kind: Secret
metadata:
  name: ai-models-artifacts
  namespace: d8-system
type: Opaque
stringData:
  accessKey: "<access-key>"
  secretKey: "<secret-key>"

Enable the module:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: ai-models
spec:
  enabled: true
  version: 1
  settings:
    logLevel: Info
    artifacts:
      bucket: ai-models
      endpoint: https://s3.example.com
      region: us-east-1
      credentialsSecretName: ai-models-artifacts
      usePathStyle: true

If object storage uses a custom CA, add ca.crt to a separate Secret in d8-system and set artifacts.caSecretName. You can also put ca.crt into the credentials Secret; the module uses it as a trust source.

The module-local Secret in d8-ai-models is rendered by Helm from data prepared by the synchronization hook. The administrator manages only the source Secret in d8-system.

Model Delivery

delivery.type selects how models in the Ready phase are attached to workloads. This setting is local to one cluster; it is not a model source and not an external catalog. If the delivery block is omitted, SharedPVC is used.

Whichever mode is selected, admission adds a scheduling gate to the Pod template of a workload that references a model, and the controller then writes the delivery template — volumes, mounts, and environment variables — into the same object. The gate follows the model reference annotation, not delivery.type.

For a Deployment this produces a second ReplicaSet: the changed Pod template makes Kubernetes create a new revision, and the superseded one — it still carries the gate but no delivery state — is deleted by the controller together with its SchedulingGated Pod, so nothing has to be cleaned up by hand. The gate is added again whenever the submitted Pod template carries no delivery state or the set of model references changes, so re-applying the source manifest — which is what GitOps does — goes through the same extra revision.

SharedPVC

SharedPVC fits clusters with storage that supports ReadWriteMany:

spec:
  settings:
    delivery:
      type: SharedPVC
      sharedPVCStorageClassName: rwx-storage-class

When sharedPVCStorageClassName is empty, storage class resolution uses:

1. global.modules.storageClass;
2. global.defaultClusterStorageClass;
3. Kubernetes default StorageClass.

The selected class must exist. The storage provisioner must then bind a ReadWriteMany PVC. Unmet prerequisites are reported at the module level as soon as the module starts — see Storage prerequisites below — and again per workload when one appears: reason SharedPVCStorageClassMissing when the class is not found, and SharedPVCStorageClassAmbiguous when more than one Kubernetes default StorageClass exists (set an explicit module or Deckhouse global storage class to make the choice deterministic).

If the provisioner cannot provision the volume, the module reports the provisioner’s own explanation in the delivery status with reason SharedPVCProvisioningFailed. Only an explanation about that exact claim is quoted, so an Event written by hand in the workload namespace cannot dictate the report. A volume that stays unprovisioned for more than 10 minutes with nothing reported about it is reported as SharedPVCProvisioningTimedOut.

Two cases are reported on their own terms instead of waiting out that timer. A class with volumeBindingMode: WaitForFirstConsumer is reported immediately as SharedPVCStorageClassDefersBinding: the cluster announces deferred binding as a Normal event rather than a failure, while for this delivery it is terminal — the consumer that would trigger binding is created only after the volume binds. A volume that bound and then lost its PersistentVolume is reported as SharedPVCClaimLost, because the data is gone rather than late.

None of these reports is latched: once the volume binds, delivery continues on its own.

A delivery that stops for a transient reason — a referenced model that loses readiness, a workload blocked by its own contract — keeps a volume that is already bound, together with the reference that holds it, so the materialized model is not downloaded again when the delivery resumes. Volumes that never bound are still reclaimed, and a workload that is deleted, loses its model reference, or moves to another delivery mode releases its volumes as before.

Storage prerequisites

The module orders volumes lazily — the first workload that references a model triggers it — so a module that installed cleanly is not yet proof that model delivery can work. The module therefore evaluates the storage prerequisites of the configured delivery mode continuously and independently of whether any model or workload exists, and reports the verdict through the d8_ai_models_storage_prerequisites_satisfied metric, the D8AIModelsStoragePrerequisitesNotSatisfied alert, and the controller log (kubectl -n d8-ai-models logs deploy/ai-models-controller -c controller | grep storage-readiness).

Unmet prerequisites do not disable the module: the catalog, sources and read APIs keep working. The module is degraded, not broken, and the verdict clears by itself once the cluster is fixed — no restart, no configuration change.

What each mode requires:

• SharedPVC — a StorageClass that can serve ReadWriteMany volumes and uses volumeBindingMode: Immediate. A class with volumeBindingMode: WaitForFirstConsumer cannot work: such a class binds a volume only after a consumer Pod exists, while this mode creates the materializer Job only after the volume is bound and holds the referencing workload Pod on a scheduling gate until then — so the volume would never get a first consumer. This is reported as StorageClassDefersBinding.
• NodeCache — a local class provided by the sds-node-configurator and sds-local-volume modules.

Reported reasons and what to fix:

ReadWriteMany capability itself is not part of this verdict: a StorageClass does not declare its access modes, so the module does not guess — RWX support is confirmed or refuted by the first provisioning attempt, and its failure is reported as described above.

A local RWO PVC is not a separate delivery mode. If the model must be kept near applications on selected nodes, use NodeCache: the module creates a node cache and exposes the model to workloads through a read-only CSI mount.

Each consumer namespace materializes the model into its own ReadWriteMany PVC, and the workload pod stays SchedulingGated until that materialization finishes.

Concurrent materializations contend for registry-read and storage-write bandwidth, so delivery.maxConcurrentMaterializations bounds how many materializer Jobs run at once cluster-wide (default 2). Raise it on clusters with more nodes or faster storage; lower it if concurrent materializations saturate the backend:

spec:
  settings:
    delivery:
      type: SharedPVC
      maxConcurrentMaterializations: 4

NodeCache

NodeCache is intended for large models and repeated model reuse by multiple workloads on the same node.

1. Enable sds-node-configurator and sds-local-volume.

2. Label cache nodes:

   d8 k label node <node-name> ai.deckhouse.io/model-cache=true

3. Label free BlockDevice objects:

   d8 k label blockdevice <block-device-name> ai.deckhouse.io/model-cache=true

4. Enable NodeCache delivery:

   spec:
     settings:
       delivery:
         type: NodeCache
         nodeCacheSize: 200Gi

By default, nodes and block devices are selected by ai.deckhouse.io/model-cache=true. If the cluster has a different labeling scheme, set delivery.nodeCacheNodeSelector and delivery.nodeCacheBlockDeviceSelector.

Check substrate state:

d8 k get blockdevices.storage.deckhouse.io -o wide
d8 k get lvmvolumegroupsets.storage.deckhouse.io
d8 k get lvmvolumegroups.storage.deckhouse.io
d8 k get localstorageclasses.storage.deckhouse.io
d8 k -n d8-ai-models get pods,pvc -l app=ai-models-node-cache-runtime -o wide

The selected disk must be free and have consumable=true.

Storage Limit

artifacts.capacityLimit sets the total budget for module-owned artifacts:

spec:
  settings:
    artifacts:
      capacityLimit: 500Gi

When the limit is set, upload gateway accepts uploads only when payload size is known. A regular curl -T sends Content-Length; multipart clients send size through /probe.

Object Storage and DMCR

The bucket from artifacts.bucket is owned by the module. Do not store unrelated data there and do not delete objects manually: the controller and DMCR keep their own references between objects, and manual deletion can break a local model copy or a later workload delivery retry.

One bucket serves one cluster. The module writes its registry to a fixed location inside the bucket and does not partition it per cluster, so two clusters pointed at the same bucket overwrite each other’s registry state. Two clusters may share one S3-compatible endpoint and one set of credentials, but each needs its own bucket. This applies to the DMZ and internal clusters of a cross-perimeter topology as well: distribution copies models between clusters over the catalog API, not through a shared bucket.

DMCR (Deckhouse Model Container Registry) is the module’s internal OCI registry. It stores prepared models as OCI artifacts on top of the configured S3-compatible bucket. The administrator configures the bucket and credentials, but does not manage OCI paths, tags, service links, or DMCR objects by hand.

A model in DMCR is not stored as a single file. The controller packages the source model files as an internal OCI ModelPack artifact without changing the model weight format. That is why one model can appear in the object storage UI as dozens or hundreds of objects. Some objects are manifests, configs, layers, and registry links; some are source upload or source mirror staging data; some are service markers that allow preparation replay and safe cleanup.

The following groups are useful for orientation in an S3-compatible storage UI. Prefix names are internal structure and are not a stable API.

Object count is not the model count. A small object can be only a service link, while one large model layer can occupy gigabytes. One layer can also be shared by several artifacts, so deleting one model does not always free exactly that model size immediately.

Deleting a Model or ClusterModel starts asynchronous cleanup:

1. The controller removes the model reference from the catalog and queues a cleanup request.
2. The DMCR cleanup helper coalesces requests, opens a maintenance gate, and waits for replica acknowledgements.
3. It then removes stale staging prefixes, aborts stale multipart uploads, and runs OCI registry garbage collection.
4. The cleanup result is published through metrics and logs.

Objects can therefore remain in the bucket right after model deletion. This is normal while alerts D8AIModelsPublicationCleanupBacklogStale and D8AIModelsPublicationCleanupFailed are not firing and the dashboard shows completed cleanup cycles.

Safe checks:

d8 k get models.ai.deckhouse.io -A
d8 k get clustermodels.ai.deckhouse.io
d8 k -n d8-ai-models get secrets -l ai.deckhouse.io/dmcr-gc-request=true
d8 k -n d8-ai-models logs deploy/dmcr -c dmcr-garbage-collection --since=2h

In logs, look for dmcr garbage collection completed: it contains deleted object count, reclaimed bytes, and deleted registry blob count. If bucket objects keep growing while cleanup requests are stuck or failed, fix the alert cause first. Manual object deletion from the bucket is allowed only as a separate emergency procedure with a verified prefix list.

Model Data Flow

The module has one model preparation path and two workload delivery paths.

Preparation reads a model source, verifies the data, and packages the source files as an internal OCI ModelPack artifact. This is not model weight conversion: GGUF stays GGUF, Safetensors stays Safetensors. DMCR stores that artifact as the local verified copy. Monitoring shows both logical model bytes and stored artifact bytes; the publication path stores raw chunk-pack layers and does not spend CPU on model-byte compression.

SharedPVC materializes the model from DMCR into a controller-owned RWX PVC in the workload namespace. The materializer Job does not use the Kubernetes API. Progress is measured by DMCR from the signed read grant issued for that Job, so the dashboard can show expected bytes, pulled bytes and throughput per materializer Job.

NodeCache materializes the model from DMCR into the node-local cache. The long-running node-cache runtime reports expected and downloaded bytes per node and artifact identity, plus cache footprint and CSI request latency.

Distribution uses a separate byte path. A consuming cluster first reads the semantic catalog, then imports selected OCI artifacts as local copies. The DMCR logs distribution pulls with consumer identity and exposes pull rate and byte metrics grouped by transfer purpose.

Cross-Perimeter Distribution

Distribution is a catalog/import plane. It exposes ClusterModel objects in the Ready phase between registry tiers and network zones. It does not change how a model is attached to a workload.

Typical topology:

1. A publishing cluster in the DMZ exposes ClusterModel objects in the Ready phase.
2. A consuming cluster in the internal perimeter imports selected models as local copies.
3. Workload delivery in the internal cluster remains SharedPVC or NodeCache.

This topology is useful when a DMZ cluster is only a distribution tier: it prepares and serves model artifacts, but has no annotated workloads. Distribution therefore remains a separate axis, not a third delivery.type value.

Both clusters run their own module instance, each with its own bucket — see Object storage and DMCR. A model exists in the internal cluster only after it is imported there; the publishing cluster does not push anything by itself.

Enable the public catalog mode on the publishing tier:

spec:
  settings:
    distribution:
      mode: PublicCatalog

After the setting is applied, the module prepares the distribution transport on the module public host:

https://ai-models.example.com/api/distribution/v1/models
https://ai-models.example.com/v2

/api/distribution/v1/models serves the semantic catalog: only ClusterModel objects in the Ready phase, without internal registry names or object UIDs. /v2 remains the OCI byte path for controller-owned copy/import workflows.

Consumer Access

Public catalog access uses Kubernetes authentication and authorization in the publishing cluster. The module does not create a separate consumer CRD.

Create a ServiceAccount for each consuming cluster, organization, or perimeter and bind it to the module distribution reader role:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: perimeter-a
  namespace: d8-ai-models
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ai-models-distribution-reader-perimeter-a
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: d8:ai-models:distribution:reader
subjects:
  - kind: ServiceAccount
    name: perimeter-a
    namespace: d8-ai-models

Issue a token in the publishing cluster and pass only the token value to the consuming administrator through a protected external channel:

d8 k -n d8-ai-models create token perimeter-a --duration=720h

For a long-lived operational credential, create a Kubernetes service-account-token Secret in the publishing cluster and read the token key after Kubernetes fills it:

apiVersion: v1
kind: Secret
metadata:
  name: perimeter-a-token
  namespace: d8-ai-models
  annotations:
    kubernetes.io/service-account.name: perimeter-a
type: kubernetes.io/service-account-token

The module does not issue or move this token automatically between clusters: that requires an external trusted channel or a secret manager. Rotate by issuing a new token in the publishing cluster and updating the consuming cluster Secret. After the Secret is updated, the controller rereads ModelCatalogSource and continues with the new token.

Revoke by deleting the RoleBinding or the ServiceAccount. Revocation closes new catalog requests and pull grants; already issued short-lived pull grants expire by TTL.

Consuming Cluster Setup

Consuming clusters describe upstream catalogs with ModelCatalogSource:

apiVersion: ai.deckhouse.io/v1alpha1
kind: ModelCatalogSource
metadata:
  name: dmz
spec:
  url: https://ai-models.dmz.example.com
  credentialsSecretName: ai-models-dmz-read
  caSecretName: ai-models-dmz-ca

The Secrets named by credentialsSecretName and caSecretName live in d8-system. The controller reads them directly for catalog refresh and pull grant exchange. The credentialsSecretName Secret contains the publishing-cluster ServiceAccount token key; caSecretName contains ca.crt and is needed only for a private CA on the external catalog. These Secrets are not copied to workload namespaces.

Models are not selected in ModuleConfig. The publishing cluster exports all ClusterModel objects in the Ready phase, and consuming users import selected models through spec.source.catalog.name.

The consuming catalog view is cluster-scoped. It shows remote catalog entries and local copies as Model or ClusterModel references, including the namespace for namespaced Model objects. This projection is available only to module management personas and does not include source URLs, Secret names, tokens, OCI repositories, tags, or blob lists.

Audit

The catalog API authenticates bearer tokens with TokenReview and authorizes requests with SubjectAccessReview: catalog list requires list on clustermodels.ai.deckhouse.io; item lookup and pull grants require get on the selected ClusterModel. It writes structured audit events for catalog_list, catalog_get, pull_grant_issued, catalog_auth_denied, and conflict/error paths with the Kubernetes username and UID. DMCR writes manifest/blob pull events with the same identity, grantID, modelName, digest, and the authorization result. Raw bearer tokens and token hashes are never written to audit.

Hugging Face Provider Catalog

Besides cluster-to-cluster catalogs (type: Catalog), a ModelCatalogSource can index a Hugging Face provider catalog. Set type: HuggingFace and a bounded policy instead of a url:

apiVersion: ai.deckhouse.io/v1alpha1
kind: ModelCatalogSource
metadata:
  name: hf-openai
spec:
  type: HuggingFace
  policy:
    allowedOrganizations:
      - openai-community
    maxIndexedEntries: 200
  # huggingFace:
  #   tokenSecretName: hf-token   # optional, for gated/private repositories

Requirements and behavior:

• The external catalog data services must be enabled (PostgreSQL and Valkey, see CONFIGURATION). When they are disabled, provider sources stay Waiting.
• policy is mandatory and must be bounded: at least one scope selector (allowedOrganizations, allowedRepositories, or allowedCollections) and maxIndexedEntries (1..10000). An unbounded source is blocked with SourcePolicyTooBroad before any provider scan.
• huggingFace.tokenSecretName is optional and names a d8-system Secret with a token key. Listing and importing public models works anonymously; gated or private repositories need the token.
• The source is indexed into PostgreSQL by the bounded sync loop as soon as it is declared, and re-indexed periodically. Large catalogs fill incrementally within the provider rate limit. The indexed entry count is in status.entryCount; health is in the Ready condition.

Provider entries stay in the durable index until imported; the index is a discovery layer, not a second source of truth. Importing an entry is the same declarative flow as any catalog source (see the user guide): create a Model or ClusterModel with spec.source.catalog.{sourceName, name}.

Browsing The Catalog

The browse surface is the modelcatalogsources/catalog-import subresource, served read-only over HTTP and authorized by kube-rbac-proxy. It is the API a frontend consumes to list sources and their entries.

Endpoints (method GET, prefix /api/catalog-import/v1):

For type: HuggingFace sources, /entries accepts bounded pagination and filter parameters: limit (default 50, max 200), offset, query (name match), org, and tag (repeatable). The response carries page.nextOffset when more entries are available. For type: Catalog sources the full snapshot is returned and pagination is ignored.

Authorization is a Kubernetes SubjectAccessReview for get on modelcatalogsources/catalog-import. That permission belongs to the module read surface: it is granted from the User access level up (access levels are cumulative) and by the rbacv2/manage view role, so a bearer token bound to any of them may browse. The binding has to be cluster-wide: the review is issued for a cluster-scoped resource, so a subject whose access level was limited to namespaces gets 403 even at the right level. Browsing is how spec.source.catalog.{sourceName, name} is picked for a Model, so it follows the read surface; declaring a provider source still requires ClusterAdmin.

API contract

Both catalog HTTP surfaces — this browse API and the cross-perimeter distribution catalog API (/api/distribution/v1) — are described by a single OpenAPI 3 contract committed at images/controller/api/catalog/openapi.yaml, which is their authoritative source of truth. A consumer can generate a client directly from it. The OCI byte path (/v2) and the upload-session dataplane (/v1/upload/*) are byte/streaming protocols and are intentionally not part of this contract.

Entry metadata

For type: HuggingFace sources, each entry carries descriptive metadata collected during synchronization from the provider listing, so a model can be evaluated before import: task (pipeline), libraryName, license, downloads, likes, createdAt, lastModified, and the pinned version (commit revision). Fields the provider does not expose for a given model are omitted. These facts are refreshed on every successful sync cycle; a field is only as fresh as the source’s last successful refresh (see the source’s lastSuccessfulRefreshTime). type: Catalog (snapshot) sources carry only what the upstream catalog exports and do not populate provider popularity fields.

Inference sizing facts

For type: HuggingFace entries, once a model has been profiled the response carries an optional sizingFacts object with the architecture and artifact facts an external inference-sizing module needs — facts only; this module computes no VRAM, GPU fit, or parallelism. It has two parts:

• model: architecture-invariant facts derived from config.json — architecture, family, model type, task, parameter count, context window, and the raw dimensions (layer count, hidden size, attention and key/value head counts, head dimension, intermediate size, vocabulary size, sliding window, MoE expert counts, encoder-decoder flag), plus a confidence map per derived fact.
• variants: one entry per weight packaging (safetensors, each GGUF quant, pytorch), each with format, quantization, precision and weight/shard byte sizes — because each variant is a distinct sizing target.

Facts are derived without downloading the weights (only config.json, tokenizer_config.json and the file-size listing are fetched) and are cached, keyed by revision. They are populated by a bounded background warm-up after sync, so sizingFacts is omitted for models not yet profiled and for gated, private or otherwise unfetchable models. Unknown fields are omitted rather than guessed.

List sources

GET /api/catalog-import/v1/sources
Authorization: Bearer <token>

{
  "apiVersion": "catalogimport.ai.deckhouse.io/v1",
  "kind": "CatalogImportSourceList",
  "items": [
    {
      "name": "hf-openai",
      "ready": "True",
      "entryCount": 200,
      "lastSuccessfulRefreshTime": "2026-06-24T21:15:35Z",
      "conditions": [
        { "type": "Ready", "status": "True", "reason": "Ready" }
      ]
    }
  ]
}

Source summary fields:

List entries

GET /api/catalog-import/v1/sources/hf-openai/entries?limit=2&offset=0
Authorization: Bearer <token>

{
  "apiVersion": "catalogimport.ai.deckhouse.io/v1",
  "kind": "CatalogImportEntryList",
  "source": { "name": "hf-openai", "ready": "True", "entryCount": 200 },
  "sourceName": "hf-openai",
  "items": [
    {
      "name": "openai-community/gpt2",
      "lifecycle": "Active",
      "artifact": { "digest": "" },
      "updatedAt": "2026-06-24T21:15:09Z",
      "source": { "type": "huggingface", "revision": "" },
      "projection": { "remoteState": "RemoteDiscovered" }
    }
  ],
  "page": { "limit": 2, "offset": 0, "nextOffset": 2 }
}

Top-level response fields:

Entry fields (items[]):

Provider entries carry minimal facts after sync (id, lifecycle, revision); rich fields such as format or parameterCount are filled in deeper, at resolve/import. Error responses use { "error": "<message>" } with the HTTP status (400 missing source name, 404 source not found, 409 source not ready, 502 provider error, 503 provider browse unavailable, 401/403 authorization).

Model storage summary

GET /api/catalog-import/v1/storage-summary
Authorization: Bearer <token>

{
  "apiVersion": "catalogimport.ai.deckhouse.io/v1",
  "kind": "CatalogImportStorageSummary",
  "summary": {
    "capacityKnown": true,
    "usageKnown": true,
    "limitBytes": 2199023255552,
    "usedBytes": 812345678901,
    "reservedBytes": 10737418240,
    "availableBytes": 1375940158411,
    "namespaced": { "usedBytes": 512345678901, "reservedBytes": 10737418240 },
    "cluster": { "usedBytes": 300000000000, "reservedBytes": 0 }
  }
}

Storage here means the module-owned artifact store that holds downloaded models, accounted by the controller’s own ledger — not the delivery volumes a workload later mounts. usedBytes covers published artifacts, reservedBytes covers downloads still in flight, and the namespaced/cluster sections attribute the same bytes to Model and ClusterModel owners respectively.

The budget comes from the module’s configured artifact capacity limit. When no limit is configured, capacityKnown is false and limitBytes/availableBytes are omitted — a consumer must render “unknown” rather than substitute a constant of its own. Occupancy is still reported in that case: no budget does not mean no accounting.

Whether the occupancy itself is a fact is a separate flag, usageKnown. It is false when storage accounting is switched off, and when its state is unavailable — the ledger was deleted or recreated empty. In both cases usedBytes, reservedBytes and the per-scope split are zero because nothing is known, not because nothing is stored, and a consumer must render “unknown” instead of “0 B”.

Recovery from an unavailable state is a job for the inventory synchronizer, which rebuilds the accounting from the models actually stored and then marks it trustworthy. Until it completes a pass, occupancy stays unknown even if a finished download has meanwhile recorded its own bytes: that record describes one model, not the store. New downloads are refused for the same reason while it lasts, which is at most one synchronization interval.

That bound holds only while the pass succeeds. A pass that keeps failing — a model whose accounting cannot be written, an unavailable API server, a revoked permission on the accounting ConfigMap — keeps occupancy unknown and new downloads refused for as long as the failure lasts, because lifting that state is what a pass does.

A confirmation, once given, is not taken back, so a pass that starts failing after one has succeeded does not refuse downloads. What it stops is the check itself: occupancy is no longer verified against the models actually stored, so a discrepancy — bytes of a finished download that never reached the ledger — is not repaired, and downloads are admitted against an occupancy that may understate the store. A refusal and an unverified occupancy call for different responses, so both are published as metrics:

The alerts D8AIModelsStorageAccountingLedgerUnconfirmed, D8AIModelsStorageInventorySyncFailing and D8AIModelsStorageInventorySyncStale fire on those facts. None of the four series are published when storage accounting is switched off: nothing is refused in that mode, so there is nothing to report. A pass writes the accounting for every model it can before it withholds confirmation, so the controller log names every model that blocks it rather than only the first.

That flag is what tells the three possible states apart, which a single capacityKnown could not:

With occupancy unknown no budget can be presented either — availableBytes is not computable — so the third state reports no budget as well.

The per-entry download verdict has two bounds worth knowing before a UI builds on it. It judges one entry, not a selection: the budget is read once per response and every entry is compared against it independently, so with 1 TiB free two 600 GiB entries both come back Feasible while the second download would be refused at reservation time. A “download selected” flow has to sum the sizes itself against the summary. The verdict is also present on entries that are already downloaded, where it describes a hypothetical re-download rather than an available action.

Frontend Exposure

When the external catalog is enabled and the module has a public HTTPS host, an Ingress exposes the browse API at https://<module-host>/api/catalog-import/v1, routed to the controller’s kube-rbac-proxy. A frontend presenting the user’s Dex/OIDC token is authorized by the same RBAC described above; no separate authentication layer is introduced.

Import is not part of this API. A frontend imports a model by creating a Model or ClusterModel through the regular Kubernetes API with the same token, which keeps native RBAC, audit, and GitOps.

Internal Model-Facts Lookup API

The module serves an in-cluster HTTP lookup for platform consumers — such as the ai-inference order controller — that need a model’s facts by its full reference. It is an internal service surface, not a public catalog: it is served on the controller’s kube-rbac-proxy at the path /api/internal/v1/ and is not exposed through a public Ingress.

GET /api/internal/v1/models/lookup?kind=ClusterModel&name=<name>
GET /api/internal/v1/models/lookup?kind=Model&name=<name>&namespace=<namespace>

The response is a single JSON ModelFacts object with the model’s readiness and descriptive facts (phase, ready, modelScope, format, supportedEndpointTypes, parameterCount, family, quantization, contextWindowTokens, sourceURL, and optional artifactSizeBytes). It carries facts only — no policy input and no admission verdict; class-policy evaluation stays in the consumer.

Error semantics distinguish absence from readiness, unlike the distribution API:

• 404 is returned only when the referenced object does not exist.
• A model that exists but is not ready returns 200 with ready: false and an explicit phase (it is never hidden behind a 404).
• A kind=Model request without a namespace is rejected with a client error distinct from 404.

The contract is part of the OpenAPI 3 document committed at images/controller/api/catalog/openapi.yaml; a consumer can generate a client from it.

Consumer access

Each request is authorized by the kube-rbac-proxy sidecar with a SubjectAccessReview against the virtual clustermodels/lookup subresource. Bind the shipped consumer role to the calling ServiceAccount:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ai-models-internal-model-lookup-ai-inference
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: d8:ai-models:internal:model-lookup
subjects:
  - kind: ServiceAccount
    name: <consumer-service-account>
    namespace: <consumer-namespace>

This authorization is coarse: a caller holding the role may look up any model’s facts (both Model and ClusterModel). Callers without it receive 403, and unauthenticated callers receive 401.

RBAC

The module uses the Deckhouse access-level model. Access levels are cumulative: each level inherits the levels below it, so the read surface is declared once at User and every higher level only adds its own write delta.

rbacv2/use and rbacv2/manage are two independent role trees, not one nested in the other: d8:use:capability:* are bound in a namespace, d8:manage:* are bound cluster-wide, and neither includes the other. Their resource lists overlap, their bindings do not.

ClusterModel and ModelCatalogSource are cluster-scoped, so the levels above reach them only through a cluster-wide binding. A ClusterAuthorizationRule that carries limitNamespaces or namespaceSelector is projected as RoleBindings, and a RoleBinding cannot grant a cluster-scoped resource: such a subject keeps reading Model in its namespaces and gets Forbidden on the two cluster-scoped kinds. Grant the access level without a namespace limit when a user has to see the cluster-wide catalog.

Upload credentials are exposed through a dedicated Role for one Secret from status.upload.secretName. The Role is created in the model namespace and is named ai-model-upload-reader-<model-name> or gets a stable hash for long names.

External Catalog Import Recovery

Catalog import stores frozen provenance: catalog source, model name, catalog revision, and remote digest. This prevents a workload from silently moving to another model version during a later reconcile.

The following failures are recoverable after the administrator fixes the source-side problem:

• CatalogAuthFailed — token expired, Secret was updated, or RBAC on the publishing cluster was fixed;
• CatalogTLSInvalid — caSecretName or ca.crt was fixed;
• CatalogSourceNotReady — the external catalog source returned to the Ready phase.

After ModelCatalogSource becomes healthy, the controller retries importing the same frozen model. ManifestInvalid, InsufficientStorage, and an invalid catalog contract are not automatic retry paths: fix the source artifact, storage limits, or the catalog specification first.

Check:

d8 k get modelcatalogsources.ai.deckhouse.io
d8 k describe modelcatalogsource <name>
d8 k -n <namespace> describe model <name>

Monitoring

Check monitoring resources:

d8 k -n d8-ai-models get podmonitor,prometheusrule

Main dashboard sections:

• Cluster overview. Shows the Model and ClusterModel inventory, objects in Publishing, Ready, and Failed phases, total prepared local copy size, managed workload count, and model references that the controller could not resolve. Start diagnostics here: non-zero Failed objects and unresolved references mean that you should drill down to a specific model or workload.
• Catalog state. Separate dashboards for namespace-scoped Model objects and cluster-scoped ClusterModel objects help locate whether the problem is in one namespace, in the shared cluster catalog, or in one selected model. Check phase, readiness, conditions, source, format, local copy size, model consumers, and workloads with unresolved delivery.
• Model preparation. Shows objects currently being prepared, upload and packaging progress, transfer throughput, completion or verification errors, and retries. If progress does not move for a long time, compare transfer throughput with DMCR/bucket state and inspect events on the related Model or ClusterModel.
• DMCR and bucket. Capacity panels show the configured limit, used, reserved, and available space for prepared local copies. Storage efficiency is shown separately: the logical model size and actual stored bytes can differ because of layer chunking, archiving, and data reuse. The cleanup queue shows pending, active, and failed cleanup requests after model deletion.
• Workload delivery. Shows which workloads are managed by the module, how many Pods are ready, which delivery mode was selected, and why. For SharedPVC, check PVC state, copy queue, and materializer Job throughput. For unknown delivery mode or unresolved references, verify model name, namespace, and permission to use the model.
• Node cache. Used for NodeCache mode: shows runtime Pods, bound PVCs, used and available space per node, cache entry count, copy throughput, materialization concurrency, and CSI mount/unmount request latency. Growing latency or low effective free space usually points to a local disk, PVC, or node-cache runtime issue.
• Catalog distribution. If cross-perimeter distribution is enabled, check public catalog request rate, pull grant issuance, API latency, and import throughput. Authorization errors or growing latency should be compared with consumer RBAC, API-server state, and audit events in the publishing cluster.

Operational Checks

Check components:

d8 k -n d8-ai-models get pods -o wide
d8 k get models.ai.deckhouse.io -A
d8 k get clustermodels.ai.deckhouse.io

Check a model:

d8 k -n <namespace> describe model <name>
d8 k get clustermodel <name> -o yaml

Useful fields:

• status.phase;
• status.conditions;
• status.artifact.digest;
• status.artifact.sizeBytes;
• status.resolved.format;
• status.resolved.supportedEndpointTypes;
• status.resolved.supportedFeatures.

Disable

When spec.enabled=false, module-owned volatile runtime resources are removed: node-cache runtime Pod/PVC, CSIDriver, LocalStorageClass, LVMVolumeGroupSet, managed LVMVolumeGroup, and StorageClass ai-models-node-cache.

Model, ClusterModel, and already prepared local model copies remain. To delete a model, delete the corresponding Model or ClusterModel; the controller finishes cleanup through a finalizer and GC request.