The module lifecycle stage: Generally available version
The module has requirements for installation
The observability module can run its own log storage based on Grafana Loki. It replaces the separate deprecated loki module.
Enable it in the observability ModuleConfig:
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
name: observability
spec:
version: 1
settings:
loki:
enabled: trueTo deliver logs into it, the log-shipper module must be enabled as well.
Modes and storage
mode |
storage.type |
Where data lives | How logs are deleted |
|---|---|---|---|
Single |
Disk |
A PersistentVolume | By disk pressure: the oldest logs go when the disk fills |
Single |
S3 |
An S3-compatible bucket | By age, after retentionPeriodHours |
Distributed |
S3 (only) |
An S3-compatible bucket | By age, after retentionPeriodHours |
Single runs one Loki instance and is the default. Distributed runs Loki in the Simple Scalable topology — separate write, read and backend targets that can be scaled independently — and requires storage.type: S3, since the targets share the store. The storage is configured once, in storage, whatever the mode.
The disk
Every stateful Loki pod — loki in Single mode, write and backend in Distributed mode — has a disk described by storage.disk. What is on it depends on storage.type:
storage.type |
What is on the disk | How to size it | Retention |
|---|---|---|---|
Disk |
Chunks, the index and the write-ahead log | For the period set by retentionPeriodHours |
By disk pressure, see LokiInsufficientDiskForRetention |
S3 |
The write-ahead log, the index cache and compactor scratch space | Small; the default is enough | By age, enforced by the compactor |
With S3 the disk is still a PersistentVolume: the write-ahead log holds the logs the ingester has accepted but not yet flushed to the bucket, and an eviction or a node failure would lose them from an ephemeral disk. storage.disk.storageClass: false replaces the PersistentVolume with an emptyDir in every mode.
A disk
settings:
loki:
enabled: true
mode: Single
storage:
type: Disk
disk:
storageClass: default
sizeGigabytes: 50
retentionPeriodHours: 168With the data on the disk retentionPeriodHours is a target rather than a limit. Loki only deletes data once the disk approaches full, evicting the oldest logs first. If that happens before logs reach the requested age, the LokiInsufficientDiskForRetention alert fires — either give Loki a bigger disk or lower the expectation.
An object store
settings:
loki:
enabled: true
mode: Single
storage:
type: S3
s3:
endpoint: https://s3.example.com
bucket: loki
region: us-east-1
accessKeyID: ...
secretAccessKey: ...
s3ForcePathStyle: true
retentionPeriodHours: 720With S3, retentionPeriodHours is enforced directly: the compactor deletes data older than that regardless of how much space is left.
Scaling out
settings:
loki:
enabled: true
mode: Distributed
storage:
type: S3
s3:
endpoint: https://s3.example.com
bucket: loki
accessKeyID: ...
secretAccessKey: ...
distributed:
write:
replicas: 3
read:
replicas: 5
backend:
replicas: 2write cannot go below three replicas: the ring is configured with a replication factor of 3.
Migrating from the loki module
The loki module is deprecated and its storage moves here. The migration is driven by you, in a fixed order, and both modules have to agree before anything is handed over.
-
Enable the built-in storage in the
observabilityModuleConfig, as shown above. Nothing is deployed yet — the module publishes a readiness signal and waits, because starting a second Loki on the same volume would fight the first one.If the
lokiModuleConfig setsnodeSelectorortolerations, copy them tosettings.loki.nodeSelectorandsettings.loki.tolerationsin the same step. The built-in storage does not read the settings of thelokimodule: without them its pods are placed automatically, and withWithDataVolumeon a node-local volume a pod that does not tolerate the taints of that node staysPending. The same applies when thelokimodule was placed automatically on dedicated nodes: it selectsnode-role.deckhouse.io/lokiand tolerates the taintdedicated.deckhouse.io=loki, while the built-in storage looks forobservabilityandsystemnodes. If the Loki nodes carry that label or taint, setsettings.loki.nodeSelectorandsettings.loki.tolerationsto match them. -
Wait for the
LokiModuleSupersededByObservabilityalert. It confirms thatobservabilityis ready to take the storage over. -
Choose what happens to the data already collected and set
migrateToObservabilityLokiin thelokiModuleConfig:apiVersion: deckhouse.io/v1alpha1 kind: ModuleConfig metadata: name: loki spec: settings: migrateToObservabilityLoki: WithDataVolumeValue Result WithDataVolumeThe existing volume is adopted, so the logs collected so far stay queryable SimpleThe new storage starts empty; the old volume is left in the cluster untouched With
storageClass: falsethere is no volume to adopt andWithDataVolumebehaves likeSimple. -
The
lokimodule stops deploying its resources,observabilitypicks the storage up and the log storage comes back. OnWithDataVolumethe volume is rebound rather than copied, so the size of the data does not affect how long this takes. -
Wait for the
LokiModuleHandoverCompletedalert and delete thelokiModuleConfig — from the cluster, and from your Git repository if the configuration is managed through GitOps.
The handover is not reversible: reverting migrateToObservabilityLoki does not bring the storage back to the loki module.
What happens to the adopted data
Data written by the old Loki uses the boltdb-shipper index with schema v11, which Loki 3.x still reads. Instead of converting it, the module keeps that period in the schema and starts a modern tsdb/v13 period from the day after the migration. Old and new logs are both queryable; the old ones eventually disappear through disk-based eviction, which always removes the oldest tables first.
The boundary day is recorded in the observability.deckhouse.io/tsdb-from annotation on the adopted PersistentVolumeClaim (d8-observability/storage-loki-0) and never recalculated. Do not edit or remove it: moving the boundary makes data already written under tsdb be looked up in the old index, and it disappears from query results.
Until the boundary day Loki runs with structured metadata disabled — it refuses to start otherwise while the active schema period is still boltdb-shipper. The module lifts the restriction on the boundary day, restarting Loki once.
Log delivery during the handover
log-shipper delivers into Loki through the ClusterLogDestination named d8-loki. The loki module deletes its d8-loki when it hands over, and this module creates its own once the storage is ready to receive, so for the duration of the takeover there is no destination at all. While that lasts, log-shipper stops reading container logs rather than dropping them: its read positions are kept on the node, and reading resumes from the same place once the destination is back. Logs are only lost if kubelet rotates a container log file away before reading resumes, so run the migration at a time of low log volume. On WithDataVolume the window lasts until the old Loki pod is gone, which its termination grace period can stretch to well over an hour.
A single failed observability release during the handover, complaining that d8-loki is owned by another release, is expected: both modules briefly claim the same name. The next reconciliation succeeds on its own.
Disk size and StorageClass during the migration
While the volume is being adopted, the claim is created with exactly the capacity of the existing volume, and storage.disk.sizeGigabytes is ignored — a claim asking for more than the volume provides would never bind. To grow the disk, resize the PersistentVolumeClaim after the migration, provided the StorageClass allows expansion:
kubectl -n d8-observability patch pvc storage-loki-0 \
-p '{"spec":{"resources":{"requests":{"storage":"100Gi"}}}}'Leave storage.disk.storageClass unset, or set it to the class of the volume being adopted. A different class makes the module recreate the StatefulSet and its claim to move to that class — and once the migration is over the volume is back on its original reclaim policy, usually Delete, so the adopted data goes with the claim.
Volumes left behind
A PersistentVolumeClaim is never deleted automatically, because the logs on it may still be wanted. One is left behind when you switch to S3, turn the storage off, or hand over with Simple. A switch to S3 does not reuse the data claim storage-loki-0 for the write-ahead log: Loki gets a new wal-loki-0, and the old claim stays behind with the logs on it. The LokiOrphanedPersistentVolumeClaim alert reports them, as well as write-ahead log claims whose workload no longer runs on S3; list and remove them once the data is no longer needed:
kubectl get pvc -A -l app=loki
kubectl get pvc -A -l observability.deckhouse.io/loki-walRecovering a stalled migration
Adoption takes one step per reconciliation and works out which step that is from the volume and the claims, so an interruption resumes by itself. If the LokiVolumeMigrationStalled alert stays on, look at the same objects the module looks at:
kubectl get pv -l observability.deckhouse.io/loki-volume=true -o yaml
kubectl get pvc -A -l app=lokiThe label marks the volume being taken over. The data is safe in any intermediate state: the volume is switched to the Retain reclaim policy, and the switch is observed, before anything is deleted. The policy it replaced is kept in the observability.deckhouse.io/original-reclaim-policy annotation on the volume, and the last step puts it back and removes the annotation — so the annotation is also what tells you the migration is still under way.
What the objects show says where it stopped:
| Volume and claims | Waiting for |
|---|---|
| No annotation, reclaim policy untouched | The volume to be protected; nothing has been done yet |
Policy Retain, the claim in d8-monitoring still exists |
That claim to go, which needs the old Loki pod to go first |
The claim in d8-monitoring is gone, claimRef still points at it |
The volume to be reserved and the new claim to be created |
claimRef points at d8-observability/storage-loki-0, not Bound |
Binding, which with WaitForFirstConsumer needs the Loki pod |
No annotation, the claim is Bound |
Nothing — the migration is finished |
To rebind the volume by hand, clear the stale binding and point it at this module’s namespace:
kubectl patch pv <name> --type merge \
-p '{"spec":{"claimRef":{"apiVersion":"v1","kind":"PersistentVolumeClaim","namespace":"d8-observability","name":"storage-loki-0","uid":null,"resourceVersion":null}}}'A claimRef without a UID marks the volume as reserved for a claim that does not exist yet, so nothing else can take it while the module creates the claim.