Automated snapshots

Automated snapshots let Stronghold create backups of integrated Raft storage on a schedule and write them either to local disk or to S3-compatible object storage.

Automated snapshots are designed for recurring backups without manual command execution. They are especially useful for production clusters where backup retention must be predictable and copies should be stored outside the cluster itself.

Consider the following when using automated snapshots:

• You can create multiple named snapshot configurations.
• Each configuration defines the schedule, retention policy, and storage type.
• Supported storage types are local and aws-s3.
• For production environments, use external object storage instead of local storage: the active node can change over time, and backups must be stored outside the system they protect.

Snapshot names use the format <file_prefix>-<N>.snap, where <N> is the creation time in Unix nanoseconds. Retention cleanup only considers files or objects with this naming format under the configured path or prefix. Two configurations of the same storage type cannot use the same destination (path_prefix and file_prefix; for aws-s3 also the same bucket and region).

Each configuration runs as a separate job. After a configuration is created or changed, its job is restarted: if there is no saved status yet, the first snapshot is taken after one tenth of interval (at most 10 minutes); otherwise it is taken once interval has elapsed since the start of the last snapshot. Subsequent snapshots are taken interval after the previous one finishes. After a failed snapshot, the next attempt is made with an exponential backoff (starting at 1 minute, doubling each time, capped at interval). Changing a configuration resets consecutive_errors.

Create or update a configuration

Use the following endpoint to create or update an automated snapshot configuration:

The endpoint requires sudo privileges. The request replaces the whole configuration: there is no partial update, so always pass all required parameters, including storage_type.

Core parameters

Additional parameters for local

Additional parameters for aws-s3

Configuration examples

Local disk

The following local-snapshot.json file creates a configuration that saves a snapshot every 5 minutes into /stronghold/data/backups, keeps 4 copies, limits the space used by snapshots to 1 GiB, and uses the main_stronghold file prefix:

{
  "interval": "5m",
  "path_prefix": "/stronghold/data/backups",
  "file_prefix": "main_stronghold",
  "retain": "4",
  "storage_type": "local",
  "local_max_space": 1073741824
}

Apply the configuration from local-snapshot.json using the following command:

d8 stronghold write sys/storage/raft/snapshot-auto/config/my-local-snapshots @local-snapshot.json

stronghold write sys/storage/raft/snapshot-auto/config/my-local-snapshots @local-snapshot.json

S3-compatible storage

The following minio-snapshot.json file stores snapshots in S3-compatible object storage:

{
  "interval": "3m",
  "path_prefix": "snapshots",
  "file_prefix": "stronghold_backup",
  "retain": "15",
  "storage_type": "aws-s3",
  "aws_s3_bucket": "my_bucket",
  "aws_s3_endpoint": "minio.domain.ru",
  "aws_s3_region": "us-east-1",
  "aws_access_key_id": "<ACCESS_KEY>",
  "aws_secret_access_key": "<SECRET_ACCESS_KEY>"
}

Apply the configuration from minio-snapshot.json using the following command:

d8 stronghold write sys/storage/raft/snapshot-auto/config/my-remote-snapshots @minio-snapshot.json

stronghold write sys/storage/raft/snapshot-auto/config/my-remote-snapshots @minio-snapshot.json

Updating an existing configuration

A configuration is replaced as a whole, so to change selected fields pass the full document with the new values (including storage_type and the required parameters of the chosen storage type):

{
  "interval": "3m",
  "path_prefix": "/stronghold/data/backups",
  "file_prefix": "main_stronghold",
  "retain": "10",
  "storage_type": "local",
  "local_max_space": 1073741824
}

Apply the modified configuration from local-snapshot-update.json using the following command:

d8 stronghold write sys/storage/raft/snapshot-auto/config/my-local-snapshots @local-snapshot-update.json

stronghold write sys/storage/raft/snapshot-auto/config/my-local-snapshots @local-snapshot-update.json

After any successful update, the job of this configuration is restarted and the schedule is recalculated.

List configurations

Use the following endpoint to list automated snapshot configurations:

Example command:

d8 stronghold list sys/storage/raft/snapshot-auto/config

stronghold list sys/storage/raft/snapshot-auto/config

Read configuration parameters

Use the following endpoint to read the parameters of an automated snapshot configuration:

Example command:

d8 stronghold read sys/storage/raft/snapshot-auto/config/my-remote-snapshots

stronghold read sys/storage/raft/snapshot-auto/config/my-remote-snapshots

For aws-s3, the response includes aws_access_key_id and aws_secret_access_key in plain text, so restrict access to this endpoint.

Delete a configuration

Use the following endpoint to delete an automated snapshot configuration:

Example command:

d8 stronghold delete sys/storage/raft/snapshot-auto/config/my-remote-snapshots

stronghold delete sys/storage/raft/snapshot-auto/config/my-remote-snapshots

Read backup status

Use the following endpoint to read the status of automated snapshots:

Example command:

d8 stronghold read sys/storage/raft/snapshot-auto/status/my-remote-snapshots

stronghold read sys/storage/raft/snapshot-auto/status/my-remote-snapshots

Main status fields:

• consecutive_errors: Number of backup errors in a row. Resets after a successfully completed snapshot.
• last_snapshot_end: End time of the last successful snapshot.
• last_snapshot_error: Text of the most recent error during snapshot.
• last_snapshot_start: Start time of the last completed snapshot.
• last_snapshot_url: Location of the last successful snapshot.
• next_snapshot_start: Next scheduled start time.
• snapshot_start: Start time of the current backup job.
• snapshot_url: Location of the currently written snapshot.