Regular disaster recovery drills

A backup that has never been restored does not guarantee recovery. Regular drills confirm that snapshots are usable, keys are available, the procedure is documented, and recovery time meets targets.

Goal

Regularly test two recovery scenarios:

• restoring a Raft snapshot into an isolated cluster;
• failing over to a DR secondary (Stronghold EE);

and record the actual recovery time (RTO) and data loss (RPO).

Frequency

Run the checks with the following frequency:

Prerequisites

To follow this example, you need:

• A Stronghold cluster with integrated Raft storage.
• An isolated restore environment: a separate network with no access to production databases, LDAP, and other systems managed by secrets engines.
• The unseal or recovery key share holders of the production cluster. After a snapshot restore, the original keys are required.
• For part 2: Stronghold EE and configured DR replication.

Part 1. Restoring a snapshot into an isolated cluster

Restore a production snapshot into an isolated test cluster and measure the recovery time.

1. Get a snapshot. Use the latest automated snapshot (Stronghold EE):

   d8 stronghold read sys/storage/raft/snapshot-auto/status/<CONFIG_NAME>

   <CONFIG_NAME> is the name of the automated snapshot configuration.

   The last_snapshot_url field points to the latest successful snapshot. Alternatively, save a snapshot manually:

   d8 stronghold operator raft snapshot save drill.snap

2. Check the snapshot integrity:

   d8 stronghold operator raft snapshot inspect -validate drill.snap

3. In the isolated environment, deploy a Stronghold cluster of the same or a newer version with integrated Raft storage (see Installing Stronghold on Linux). Initialize and unseal it, and log in with the temporary root token:

   d8 stronghold operator init
   d8 stronghold operator unseal
   d8 stronghold login <TEMPORARY_ROOT_TOKEN>

   <TEMPORARY_ROOT_TOKEN> is the initial root token printed by operator init.

   If the production cluster uses auto-unseal (KMS, HSM or another seal), configure the same seal stanza on the test cluster and give it access to the same key. The snapshot contains the barrier keys encrypted by the production seal (core/hsm/barrier-unseal-keys), and after the restore the test cluster decrypts them through its own seal. With a different key the unseal fails, and snapshot restore without -force refuses such a snapshot. With auto-unseal, the cluster unseals itself after the restore, and the step with the manual unseal below is not needed.

4. Record the restore start time and restore the snapshot:

   d8 stronghold operator raft snapshot restore -force drill.snap

5. With a Shamir seal, unseal the cluster with the original keys of the production cluster:

   d8 stronghold operator unseal

6. Log in to the restored cluster. The restore replaced the token store with the data from the snapshot, so the temporary root token is no longer valid. Log in with a token from the production cluster that has permissions to read the checked data:

   d8 stronghold login

   If no such token is available, generate a root token on the test cluster. The holders of the production unseal key shares (recovery key shares with auto-unseal) take part in the generation:

   d8 stronghold operator generate-root -init
   d8 stronghold operator generate-root
   d8 stronghold operator generate-root -decode=<ENCODED_TOKEN> -otp=<OTP>
   d8 stronghold login

   The first command starts the generation and prints the OTP. Run the second command once for each key share holder: it asks for a key share. After the last share, the command prints the encoded token, <ENCODED_TOKEN>. The third command decodes it with <OTP> and prints the root token. Enter it when d8 stronghold login asks for a token.

7. Check the data and record the end time:

   d8 stronghold status
   d8 stronghold secrets list
   d8 stronghold auth list
   d8 stronghold policy list
   d8 stronghold kv get -mount=secret <CONTROL_SECRET_PATH>

   <CONTROL_SECRET_PATH> is the path to a control secret with a timestamp. Create the control secret in the production cluster in advance and update its timestamp regularly. Use it to determine the RPO: the RPO is the difference between the restore start time and the timestamp in the restored secret.

8. Destroy the environment and all snapshot copies in it.

Part 2. Planned failover to a DR secondary

Switch clients to the DR secondary and return the former primary to the secondary role.

Perform the failover in an agreed maintenance window. Do not promote the DR secondary while the primary is active: you would end up with two primaries.

1. Make sure the secondary has caught up with the primary. Read last_wal from the replication status of the primary and last_remote_wal from the replication status of the secondary:

   d8 stronghold read -address="${PRIMARY_ADDR}" -field=last_wal sys/replication/dr/status
   d8 stronghold read -address="${SECONDARY_ADDR}" -field=last_remote_wal sys/replication/dr/status

   The secondary has caught up when the two values are equal. PRIMARY_ADDR and SECONDARY_ADDR are the addresses of the primary and the secondary.

2. Demote the current primary:

   d8 stronghold write -address="${PRIMARY_ADDR}" -force sys/replication/dr/primary/demote

3. Generate a DR operation token on the secondary using the key share procedure and promote the secondary. The commands are given in Promote a DR secondary.

4. Switch clients to the new primary (DNS record or load balancer) and record the time.

5. Check operation: OIDC login, reading secrets, issuing dynamic credentials and certificates.

6. Return the former primary to the secondary role. On the new primary, issue an activation token:

   d8 stronghold write -address="${SECONDARY_ADDR}" sys/replication/dr/primary/secondary-token id=dr-old-primary

   The command prints the activation token, <ACTIVATION_TOKEN>.

7. On the former primary, which is now a DR secondary, generate a DR operation token. The update-primary call on a DR secondary requires this token. The commands use the sys/replication/dr/secondary/generate-operation-token/attempt and sys/replication/dr/secondary/generate-operation-token/update endpoints:

   d8 stronghold operator generate-root -address="${PRIMARY_ADDR}" -dr-token -init
   d8 stronghold operator generate-root -address="${PRIMARY_ADDR}" -dr-token
   d8 stronghold operator generate-root -address="${PRIMARY_ADDR}" -dr-token -decode=<ENCODED_TOKEN> -otp=<OTP>

   The first command starts the generation and prints the OTP. Run the second command once for each holder of an unseal key share (a recovery key share with auto-unseal): it asks for a key share. After the last share, the command prints the encoded token, <ENCODED_TOKEN>. The third command decodes it with <OTP> and prints the DR operation token, <DR_OPERATION_TOKEN>.

   By default, the endpoints for generating a DR operation token require authentication. To call them without a token, add generate-operation-token to the enable_unauthenticated_access parameter of the server configuration.

8. Connect the former primary to the new primary:

   d8 stronghold write -address="${PRIMARY_ADDR}" sys/replication/dr/secondary/update-primary token=<ACTIVATION_TOKEN> dr_operation_token=<DR_OPERATION_TOKEN>

9. If needed, fail back with the same steps.

Drill checklist

After the drill, check the following items:

• The snapshot passed inspect -validate.
• Key share holders are available, and the quorum was gathered in acceptable time.
• The restored cluster was unsealed with the original keys.
• The control secret was read, and the RPO was measured.
• Auth methods, policies, and mounts are present as expected.
• For DR: the secondary was promoted, clients were switched, and the former primary was connected as a secondary.
• The recovery time (RTO) was measured and compared with the target.
• The isolated environment and snapshot copies were destroyed.
• Issues found were added to the recovery runbook.

Verification

A drill is successful if the actual RTO and RPO do not exceed the targets and all checklist items are done. Keep a drill record: date, participants, Stronghold version, snapshot used, measured RTO and RPO, and issues found.

Cleanup

After the drill, delete the test environment and the data copies:

1. Delete the isolated cluster and its storage.
2. Delete local snapshot copies: shred -u drill.snap.
3. Revoke tokens issued for the drill on the production cluster.

Additional resources

• Backups
• Inspecting a snapshot
• Restoring from a snapshot
• Automated snapshots
• Disaster recovery