The module lifecycle stage: Generally available version

The module has requirements for installation

Data Backup

Deckhouse Optics (DOP) provides the ability to create a data backup by copying the contents of S3 buckets. In addition to metrics, logs, and traces that are stored in S3 during normal platform operation, DOP also automatically creates a backup of the PostgreSQL database and saves its dump to S3.

Thus, to perform a full backup of the platform data, you need to copy the contents of the following S3 buckets:

  • mimir — for metrics;
  • loki — for logs, exists only when the logs storage is enabled (storage.logs.enabled);
  • tempo — for traces, exists only when the traces storage is enabled (storage.traces.enabled);
  • backup — for PostgreSQL database dumps.

PostgreSQL backup settings are defined in the ui.postgres.backup section of the DOP configuration. If an external S3 is set in ui.postgres.backup.s3, the dumps are stored there instead of the backup bucket.

The commands below skip the loki and tempo buckets when they do not exist.

Metrics Backup

Metrics are stored in an S3 bucket in the form of TSDB blocks. To back them up, it is sufficient to copy the contents of the bucket — for example, using the rclone utility (described below).

⚠️ Note: Backup does not guarantee the preservation of all metrics. The last two hours of metrics may reside only in the ingester memory and not yet be flushed to S3. This should be considered when estimating acceptable data loss.

Block Structure Details

  • TSDB blocks are immutable and have unique names. This allows safe use of simple copying tools without interrupting ingestion.
  • A block is considered complete only after the meta.json file is written (created last).
  • Two types of blocks may be present in the backup:
    • Complete blocks (with meta.json) are ready for recovery.
    • Incomplete blocks (without meta.json, also known as partial) are not used in queries, are automatically deleted over time by the system, and may be present in the backup.
  • If a block contains meta.json but lacks the chunks or index data directories, the block is considered corrupted. These blocks appear only due to copying failures and are resolved by re-running the backup process.

Traces Backup

Traces are stored in an S3 bucket in the form of Tempo blocks. To back them up, it is sufficient to copy the contents of the tempo bucket — for example, using the rclone utility (described below). The same approach and considerations as for metrics backup apply.

⚠️ Note: Backup does not guarantee the preservation of all traces. Recent traces may reside only in the ingester memory and not yet be flushed to S3. This should be considered when estimating acceptable data loss.

S3 Bucket Backup Methods

S3 buckets can be backed up using the following approaches:

  • Use utilities such as rclone or minio-client to synchronize S3 bucket contents with another local or S3-compatible storage.
  • Mount S3 buckets to the local file system with tools like s3fs or geesefs, then use your company’s standard backup system to copy data.

Using geesefs

  1. Install geesefs on the server where the buckets will be mounted:

    curl -L https://github.com/yandex-cloud/geesefs/releases/download/v0.43.0/geesefs-linux-amd64 -o /usr/local/bin/geesefs
chmod +x /usr/local/bin/geesefs

  2. Create mount points for the S3 buckets:

    mkdir -p /mnt/dop-backup/mimir /mnt/dop-backup/backup

    If the logs or traces storage is enabled, also create /mnt/dop-backup/loki or /mnt/dop-backup/tempo.

  3. Obtain a credentials file for accessing S3 buckets. On the Kubernetes control-plane node, run:

    kubectl -n d8-observability-platform get secrets backup-s3 loki-s3 mimir-s3 tempo-s3 --ignore-not-found -o json | jq -r '.items[] | reduce . as $elt ({}; .[$elt.metadata.name|sub("-s3$"; "")] += [($elt.data | map_values(@base64d) | with_entries(.key |= ascii_downcase) | to_entries[] | "\(.key) = \(.value)")]) | to_entries[] | "[\(.key)]\n\(.value|join("\n"))"'

    Save the output as /etc/dop-s3-credentials on your server.

  4. Generate /etc/fstab entries. On the Kubernetes control-plane node, run:

    kubectl -n d8-observability-platform get cm backup-s3 loki-s3 mimir-s3 tempo-s3 --ignore-not-found -o json | jq -r \
  --arg endpoint $(kubectl get mc observability-platform -o json | jq -r '"https://s3." + .spec.settings.general.baseDomain') \
  '.items[] | (.metadata.name|sub("-s3$"; "")) as $name |
    "\(.data.BUCKET_NAME) /mnt/dop-backup/\($name) fuse.geesefs _netdev,allow_other,--file-mode=0644,--dir-mode=0755,--shared-config=/etc/dop-s3-credentials,--profile=\($name),--endpoint=\($endpoint) 0 0"'

    Save the output to /etc/fstab.

  5. Mount the buckets:

    mount -a

  6. Verify that the S3 buckets are mounted correctly:

    ls -l /mnt/dop-backup/backup/postgres-backup

  7. Perform backups using your organization’s standard backup tools at the required frequency.

Using rclone

  1. Install rclone:

    curl -L https://github.com/rclone/rclone/releases/download/v1.69.1/rclone-v1.69.1-linux-amd64.zip -o rclone.zip
unzip -p rclone.zip rclone-*-linux-amd64/rclone | sudo tee /usr/local/bin/rclone > /dev/null
sudo chmod +x /usr/local/bin/rclone
rm rclone.zip

  2. Generate the rclone.conf configuration file. On the Kubernetes control-plane node, run:

    kubectl -n d8-observability-platform get secrets backup-s3 loki-s3 mimir-s3 tempo-s3 --ignore-not-found -o json | jq -r \
  --arg endpoint $(kubectl get mc observability-platform -o json | jq -r '"https://s3." + .spec.settings.general.baseDomain') \
  --argjson buckets $(kubectl -n d8-observability-platform get cm backup-s3 loki-s3 mimir-s3 tempo-s3 --ignore-not-found -o json | jq -cM 'reduce .items[] as $elt ({}; .[$elt.metadata.name] = $elt.data.BUCKET_NAME)') \
  '.items[] | reduce . as $elt ({};
     .[$elt.metadata.name] += [($elt.data | map_values(@base64d)
       | with_entries(.key |= ascii_downcase) | with_entries(.key |= sub("^aws_"; ""))
       | . += {type: "s3", provider: "Ceph", endpoint: $endpoint}
       | to_entries[] | "\(.key) = \(.value)")]
     | .[($elt.metadata.name|sub("-s3$"; ""))] = ["type = alias", "remote = " + ($elt.metadata.name + ":" + $buckets[$elt.metadata.name])])
   | to_entries[] | "[\(.key)]\n\(.value|join("\n"))\n"'

  3. Save the output as rclone.conf on the backup server.

  4. Verify access to the bucket:

    rclone --config rclone.conf ls backup:postgres-backup

  5. Use rclone sync or rclone copy commands to perform backups.

Example: Metrics Backup with rclone

rclone --config rclone.conf sync -v --delete-before --exclude-if-present deletion-mark.json --exclude '*/markers/*' --exclude '*/bucket-index.json.gz' mimir: /backup/prod/mimir/

This command synchronizes the bucket with a local backup directory, excluding marker files, bucket indexes, and blocks marked for deletion. Any excluded files still present locally but not in the source will be removed.

Data Recovery

Restoring metrics, logs, and traces involves loading the backup contents back into the appropriate S3 buckets. You can use mounted buckets or rclone in the same way as during backup.

First, all blocks except meta.json are uploaded. Then, meta.json files are uploaded in a separate step. This ensures that only complete blocks are seen and processed by the system.

Example:

rclone --config rclone.conf sync /backup/prod/mimir/ mimir: --exclude '*/meta.json'
rclone --config rclone.conf sync /backup/prod/mimir/ mimir: --include '*/meta.json'

PostgreSQL Database Recovery

This procedure applies when the observability-platform module is deployed with an internal PostgreSQL database. If you are using an external database (.spec.settings.ui.postgres.mode: External), follow your DB provider’s recovery instructions.

  1. Stop the components that write to the database, and suspend the backup CronJob. Save the current number of replicas to restore it later:

    kubectl -n d8-observability-platform get deploy backend backend-scheduler alertgate-api alertgate-receiver alertgate-sender reconciler rsm-engine rsm-engine-discovery \
  --ignore-not-found -o jsonpath='{range .items[*]}{.metadata.name} {.spec.replicas}{"\n"}{end}' > dop-replicas.txt
cut -d' ' -f1 dop-replicas.txt | xargs kubectl -n d8-observability-platform scale deploy --replicas=0
kubectl -n d8-observability-platform patch cronjob postgres-backup --type=merge -p '{"spec":{"suspend":true}}'

    If PostgreSQL backup is disabled (ui.postgres.backup.enabled: false), there is no postgres-backup CronJob: skip the patch command here and in step 6.

    Wait until the pods of these components are terminated:

    kubectl -n d8-observability-platform get po -l 'app in (backend,backend-scheduler,alertgate-api,alertgate-receiver,alertgate-sender,reconciler,rsm-engine,rsm-engine-discovery)'

    Suspending the CronJob does not stop a backup that is already running. Wait until the command below prints nothing:

    kubectl -n d8-observability-platform get cronjob postgres-backup -o jsonpath='{.status.active}'

  2. Ensure that you have a database dump of the correct size. Copy it to the host where you run kubectl.

  3. Clear the public schema:

    kubectl -n d8-observability-platform exec -i $(kubectl -n d8-observability-platform get po -l cnpg.internal.managed.deckhouse.io/instanceRole=primary -o name) -c postgres -- \
  psql -v ON_ERROR_STOP=1 -c "DROP SCHEMA public CASCADE;" -c "CREATE SCHEMA public AUTHORIZATION pg_database_owner;" -c "GRANT USAGE ON SCHEMA public TO PUBLIC;" dop

  4. Restore the database from the dump:

    zcat dop-202504211200.dump.gz | kubectl -n d8-observability-platform exec -i $(kubectl -n d8-observability-platform get po -l cnpg.internal.managed.deckhouse.io/instanceRole=primary -o name) -c postgres -- \
  psql -q -v ON_ERROR_STOP=1 dop

  5. Check the restored tables:

    kubectl -n d8-observability-platform exec -i $(kubectl -n d8-observability-platform get po -l cnpg.internal.managed.deckhouse.io/instanceRole=primary -o name) -c postgres -- \
  psql -tAc "SELECT tableowner, count(*) FROM pg_tables WHERE schemaname = 'public' GROUP BY tableowner;" dop

  6. Start the components again and resume the backup CronJob:

    while read -r name replicas; do kubectl -n d8-observability-platform scale deploy "$name" --replicas="$replicas"; done < dop-replicas.txt
kubectl -n d8-observability-platform patch cronjob postgres-backup --type=merge -p '{"spec":{"suspend":false}}'

  7. Verify that all components are running:

    kubectl -n d8-observability-platform get po -l 'app in (backend,backend-scheduler,alertgate-api,alertgate-receiver,alertgate-sender,reconciler,rsm-engine,rsm-engine-discovery)'

  8. Check that the web interface is accessible.