The module lifecycle stage: Generally available version

The module has requirements for installation

The observability module can run its own log storage based on Grafana Loki. It replaces the separate deprecated loki module.

Enable it in the observability ModuleConfig:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: observability
spec:
  version: 1
  settings:
    loki:
      enabled: true

To deliver logs into it, the log-shipper module must be enabled as well.

Modes and storage

mode storage.type Where data lives How logs are deleted
Single Disk A PersistentVolume By disk pressure: the oldest logs go when the disk fills
Single S3 An S3-compatible bucket By age, after retentionPeriodHours
Distributed S3 (only) An S3-compatible bucket By age, after retentionPeriodHours

Single runs one Loki instance and is the default. Distributed runs Loki in the Simple Scalable topology — separate write, read and backend targets that can be scaled independently — and requires storage.type: S3, since the targets share the store. The storage is configured once, in storage, whatever the mode.

The disk

Every stateful Loki pod — loki in Single mode, write and backend in Distributed mode — has a disk described by storage.disk. What is on it depends on storage.type:

storage.type What is on the disk How to size it Retention
Disk Chunks, the index and the write-ahead log For the period set by retentionPeriodHours By disk pressure, see LokiInsufficientDiskForRetention
S3 The write-ahead log, the index cache and compactor scratch space Small; the default is enough By age, enforced by the compactor

With S3 the disk is still a PersistentVolume: the write-ahead log holds the logs the ingester has accepted but not yet flushed to the bucket, and an eviction or a node failure would lose them from an ephemeral disk. storage.disk.storageClass: false replaces the PersistentVolume with an emptyDir in every mode.

A disk

settings:
  loki:
    enabled: true
    mode: Single
    storage:
      type: Disk
      disk:
        storageClass: default
        sizeGigabytes: 50
    retentionPeriodHours: 168

With the data on the disk retentionPeriodHours is a target rather than a limit. Loki only deletes data once the disk approaches full, evicting the oldest logs first. If that happens before logs reach the requested age, the LokiInsufficientDiskForRetention alert fires — either give Loki a bigger disk or lower the expectation.

An object store

settings:
  loki:
    enabled: true
    mode: Single
    storage:
      type: S3
      s3:
        endpoint: https://s3.example.com
        bucket: loki
        region: us-east-1
        accessKeyID: ...
        secretAccessKey: ...
        s3ForcePathStyle: true
    retentionPeriodHours: 720

With S3, retentionPeriodHours is enforced directly: the compactor deletes data older than that regardless of how much space is left.

Scaling out

settings:
  loki:
    enabled: true
    mode: Distributed
    storage:
      type: S3
      s3:
        endpoint: https://s3.example.com
        bucket: loki
        accessKeyID: ...
        secretAccessKey: ...
    distributed:
      write:
        replicas: 3
      read:
        replicas: 5
      backend:
        replicas: 2

write cannot go below three replicas: the ring is configured with a replication factor of 3.

Migrating from the loki module

The loki module is deprecated and its storage moves here. The migration is driven by you, in a fixed order, and both modules have to agree before anything is handed over.

  1. Enable the built-in storage in the observability ModuleConfig, as shown above. Nothing is deployed yet — the module publishes a readiness signal and waits, because starting a second Loki on the same volume would fight the first one.

    If the loki ModuleConfig sets nodeSelector or tolerations, copy them to settings.loki.nodeSelector and settings.loki.tolerations in the same step. The built-in storage does not read the settings of the loki module: without them its pods are placed automatically, and with WithDataVolume on a node-local volume a pod that does not tolerate the taints of that node stays Pending. The same applies when the loki module was placed automatically on dedicated nodes: it selects node-role.deckhouse.io/loki and tolerates the taint dedicated.deckhouse.io=loki, while the built-in storage looks for observability and system nodes. If the Loki nodes carry that label or taint, set settings.loki.nodeSelector and settings.loki.tolerations to match them.

  2. Wait for the LokiModuleSupersededByObservability alert. It confirms that observability is ready to take the storage over.

  3. Choose what happens to the data already collected and set migrateToObservabilityLoki in the loki ModuleConfig:

    apiVersion: deckhouse.io/v1alpha1
    kind: ModuleConfig
    metadata:
      name: loki
    spec:
      settings:
        migrateToObservabilityLoki: WithDataVolume
    Value Result
    WithDataVolume The existing volume is adopted, so the logs collected so far stay queryable
    Simple The new storage starts empty; the old volume is left in the cluster untouched

    With storageClass: false there is no volume to adopt and WithDataVolume behaves like Simple.

  4. The loki module stops deploying its resources, observability picks the storage up and the log storage comes back. On WithDataVolume the volume is rebound rather than copied, so the size of the data does not affect how long this takes.

  5. Wait for the LokiModuleHandoverCompleted alert and delete the loki ModuleConfig — from the cluster, and from your Git repository if the configuration is managed through GitOps.

The handover is not reversible: reverting migrateToObservabilityLoki does not bring the storage back to the loki module.

What happens to the adopted data

Data written by the old Loki uses the boltdb-shipper index with schema v11, which Loki 3.x still reads. Instead of converting it, the module keeps that period in the schema and starts a modern tsdb/v13 period from the day after the migration. Old and new logs are both queryable; the old ones eventually disappear through disk-based eviction, which always removes the oldest tables first.

The boundary day is recorded in the observability.deckhouse.io/tsdb-from annotation on the adopted PersistentVolumeClaim (d8-observability/storage-loki-0) and never recalculated. Do not edit or remove it: moving the boundary makes data already written under tsdb be looked up in the old index, and it disappears from query results.

Until the boundary day Loki runs with structured metadata disabled — it refuses to start otherwise while the active schema period is still boltdb-shipper. The module lifts the restriction on the boundary day, restarting Loki once.

Log delivery during the handover

log-shipper delivers into Loki through the ClusterLogDestination named d8-loki. The loki module deletes its d8-loki when it hands over, and this module creates its own once the storage is ready to receive, so for the duration of the takeover there is no destination at all. While that lasts, log-shipper stops reading container logs rather than dropping them: its read positions are kept on the node, and reading resumes from the same place once the destination is back. Logs are only lost if kubelet rotates a container log file away before reading resumes, so run the migration at a time of low log volume. On WithDataVolume the window lasts until the old Loki pod is gone, which its termination grace period can stretch to well over an hour.

A single failed observability release during the handover, complaining that d8-loki is owned by another release, is expected: both modules briefly claim the same name. The next reconciliation succeeds on its own.

Disk size and StorageClass during the migration

While the volume is being adopted, the claim is created with exactly the capacity of the existing volume, and storage.disk.sizeGigabytes is ignored — a claim asking for more than the volume provides would never bind. To grow the disk, resize the PersistentVolumeClaim after the migration, provided the StorageClass allows expansion:

kubectl -n d8-observability patch pvc storage-loki-0 \
  -p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'

Leave storage.disk.storageClass unset, or set it to the class of the volume being adopted. A different class makes the module recreate the StatefulSet and its claim to move to that class — and once the migration is over the volume is back on its original reclaim policy, usually Delete, so the adopted data goes with the claim.

Volumes left behind

A PersistentVolumeClaim is never deleted automatically, because the logs on it may still be wanted. One is left behind when you switch to S3, turn the storage off, or hand over with Simple. A switch to S3 does not reuse the data claim storage-loki-0 for the write-ahead log: Loki gets a new wal-loki-0, and the old claim stays behind with the logs on it. The LokiOrphanedPersistentVolumeClaim alert reports them, as well as write-ahead log claims whose workload no longer runs on S3; list and remove them once the data is no longer needed:

kubectl get pvc -A -l app=loki
kubectl get pvc -A -l observability.deckhouse.io/loki-wal

Recovering a stalled migration

Adoption takes one step per reconciliation and works out which step that is from the volume and the claims, so an interruption resumes by itself. If the LokiVolumeMigrationStalled alert stays on, look at the same objects the module looks at:

kubectl get pv -l observability.deckhouse.io/loki-volume=true -o yaml
kubectl get pvc -A -l app=loki

The label marks the volume being taken over. The data is safe in any intermediate state: the volume is switched to the Retain reclaim policy, and the switch is observed, before anything is deleted. The policy it replaced is kept in the observability.deckhouse.io/original-reclaim-policy annotation on the volume, and the last step puts it back and removes the annotation — so the annotation is also what tells you the migration is still under way.

What the objects show says where it stopped:

Volume and claims Waiting for
No annotation, reclaim policy untouched The volume to be protected; nothing has been done yet
Policy Retain, the claim in d8-monitoring still exists That claim to go, which needs the old Loki pod to go first
The claim in d8-monitoring is gone, claimRef still points at it The volume to be reserved and the new claim to be created
claimRef points at d8-observability/storage-loki-0, not Bound Binding, which with WaitForFirstConsumer needs the Loki pod
No annotation, the claim is Bound Nothing — the migration is finished

To rebind the volume by hand, clear the stale binding and point it at this module’s namespace:

kubectl patch pv <name> --type merge \
  -p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","namespace":"d8-observability","name":"storage-loki-0","uid":null,"resourceVersion":null}}}'

A claimRef without a UID marks the volume as reserved for a claim that does not exist yet, so nothing else can take it while the module creates the claim.