Automated snapshots

Automated snapshots let Stronghold create backups of integrated Raft storage on a schedule and write them either to local disk or to S3-compatible object storage.

Automated snapshots are designed for recurring backups without manual command execution. They are especially useful for production clusters where backup retention must be predictable and copies should be stored outside the cluster itself.

Consider the following when using automated snapshots:

• You can create multiple named snapshot configurations.
• Each configuration defines the schedule, retention policy, and storage type.
• Supported storage types are local and aws-s3.
• For production environments, local storage is usually less suitable than external object storage: the active node can change over time, and backups are safer when stored outside the system they protect.

Snapshot names use the format <file_prefix>_<RFC3339_timestamp>.tar.gz. Retention cleanup only considers files or objects with this naming format under the configured path or prefix.

The snapshot manager checks configurations once a minute. After creating or updating a configuration, the next snapshot is scheduled immediately; subsequent snapshots are scheduled from the start time of the previous snapshot plus the configured interval.

Server configuration (S3 upload pool)

Optional top-level keys in the Stronghold server HCL configuration tune background S3 uploads for automated snapshots of Raft storage. These are set in the server configuration file used to start a specific node: stronghold server -config config.hcl.

Example:

snapshot_auto_s3_upload_workers = 4
snapshot_auto_upload_pool_shutdown = "90s"

Increase the number of background upload processes if you have many S3 snapshot configurations and sufficient network and CPU capacity; decrease it if you hit S3 rate limits or have limited bandwidth.

Create or update a configuration

The endpoint requires sudo privileges.

Core parameters

Additional parameters for local

Additional parameters for aws-s3

Retries reuse the temporary copy from the failed upload. They stop when the configured maximum is reached, when retries are disabled, when the temporary file is no longer available, or when the next scheduled snapshot time is reached. A retry attempt must also finish at least one minute before the next scheduled snapshot; otherwise the attempt is considered failed and the next one is scheduled according to the backoff schedule. Retry failures update last_snapshot_error in the status, but do not increment consecutive_errors beyond the original failed scheduled snapshot.

Configuration examples

Local disk

The following local-snapshot.json file creates a configuration that saves a snapshot every 5 minutes into /stronghold/data/backups, keeps 4 copies, and uses the main_stronghold file prefix:

{
  "interval": "5m",
  "path_prefix": "/stronghold/data/backups",
  "file_prefix": "main_stronghold",
  "retain": "4",
  "storage_type": "local"
}

Apply the configuration from local-snapshot.json using the following command:

d8 stronghold write sys/storage/raft/snapshot-auto/config/my-local-snapshots @local-snapshot.json

stronghold write sys/storage/raft/snapshot-auto/config/my-local-snapshots @local-snapshot.json

S3-compatible storage

The following minio-snapshot.json file stores snapshots in S3-compatible object storage:

{
  "interval": "3m",
  "path_prefix": "snapshots",
  "file_prefix": "stronghold_backup",
  "retain": "15",
  "storage_type": "aws-s3",
  "aws_s3_bucket": "my_bucket",
  "aws_s3_endpoint": "minio.domain.ru",
  "aws_access_key_id": "<ACCESS_KEY>",
  "aws_secret_access_key": "<SECRET_ACCESS_KEY>",
  "retry_enabled": true,
  "retry_max_attempts": 5
}

Apply the configuration from minio-snapshot.json using the following command:

d8 stronghold write sys/storage/raft/snapshot-auto/config/my-remote-snapshots @minio-snapshot.json

stronghold write sys/storage/raft/snapshot-auto/config/my-remote-snapshots @minio-snapshot.json

Updating an existing configuration

To modify only selected fields, provide a partial JSON document:

{
  "interval": "3m",
  "retain": "10"
}

Apply the modified configuration from local-snapshot-update.json using the following command:

d8 stronghold write sys/storage/raft/snapshot-auto/config/my-local-snapshots @local-snapshot-update.json

stronghold write sys/storage/raft/snapshot-auto/config/my-local-snapshots @local-snapshot-update.json

After any successful update, the next snapshot is scheduled immediately.

List configurations

Example command:

d8 stronghold list sys/storage/raft/snapshot-auto/config

stronghold list sys/storage/raft/snapshot-auto/config

Read configuration parameters

Example command:

d8 stronghold read sys/storage/raft/snapshot-auto/config/my-remote-snapshots

stronghold read sys/storage/raft/snapshot-auto/config/my-remote-snapshots

For aws-s3, the response does not expose aws_access_key_id or aws_secret_access_key.

Delete a configuration

Example command:

d8 stronghold delete sys/storage/raft/snapshot-auto/config/my-remote-snapshots

stronghold delete sys/storage/raft/snapshot-auto/config/my-remote-snapshots

Read backup status

Example command:

d8 stronghold read sys/storage/raft/snapshot-auto/status/my-remote-snapshots

stronghold read sys/storage/raft/snapshot-auto/status/my-remote-snapshots

Main status fields:

• consecutive_errors: Number of backup errors in a row. Resets after a successfully completed snapshot.
• last_snapshot_end: End time of the last successful snapshot.
• last_snapshot_error: Text of the most recent error during snapshot.
• last_snapshot_start: Start time of the last completed snapshot.
• last_snapshot_url: Location of the last successful snapshot.
• last_rotation_error: Text of the last error during retention cleanup, or n/a if cleanup succeeded. A cleanup error does not mark the snapshot itself as failed.
• next_snapshot_start: Next scheduled start time.
• snapshot_start: Start time of the current backup job.
• snapshot_url: Location of the currently written snapshot.

Additional fields for aws-s3

• retry_in_progress: Flag indicating that a retry of a failed S3 upload is in progress.
• retry_attempt: Number of the current retry attempt. 0 if retries are not active.