Disaster recovery

Disaster Recovery (DR) replication keeps a hot standby cluster — a full copy of the primary’s storage together with local data such as tokens and leases. A DR secondary does not serve clients (beyond unseal and a small internal set) and waits for a promote to take over when the primary fails.

Before you start

Before you configure DR replication, prepare the following:

• Make sure both clusters run Stronghold EE with integrated Raft storage and that replication is enabled. For details, refer to Overview.
• Make sure the primary cluster port is reachable from the secondary and that you have the primary CA certificate for TLS.
• Prepare a token with permissions for sys/replication/* on the primary.
• Prepare the tokens for a promote while the primary is available. For details, refer to “Preparing for a failover”.

In the examples below ${PRIMARY_ADDR} and ${SECONDARY_ADDR} are the API addresses of the clusters, and ${VAULT_TOKEN} is a token authorized for sys/replication/*.

Step 1. Enable the DR primary

On the primary cluster, enable DR replication in the primary mode:

d8 stronghold write -force sys/replication/dr/primary/enable

stronghold write -force sys/replication/dr/primary/enable

Step 2. Create an activation token for the DR secondary

On the primary cluster, create an activation token for the DR secondary:

d8 stronghold write sys/replication/dr/primary/secondary-token id=dr-1 ttl=24h

stronghold write sys/replication/dr/primary/secondary-token id=dr-1 ttl=24h

The id parameter is required; ttl defaults to 30m. The command returns a wrapping token in the wrap_info.token field — pass exactly this token to the secondary.

Step 3. Enable the DR secondary

On the secondary cluster, enable DR replication in the secondary mode with the activation token:

curl --header "X-Vault-Token: ${VAULT_TOKEN}" --request POST --data @dr-secondary-enable.json "${SECONDARY_ADDR}/v1/sys/replication/dr/secondary/enable"

Example dr-secondary-enable.json:

{
  "token": "<WRAPPING_TOKEN>",
  "primary_api_addr": "<PRIMARY_API_ADDR>",
  "ca_file": "<PRIMARY_CA_FILE>"
}

In the example:

• <WRAPPING_TOKEN>: Wrapping token from the wrap_info.token field returned in step 2
• <PRIMARY_API_ADDR>: API address of the primary cluster
• <PRIMARY_CA_FILE>: Path to the primary CA certificate in PEM format on the secondary server

For self-signed environments, ca_file or ca_path (the primary CA in PEM format, read on the secondary server) is required. The DR secondary copies the entire storage, including local data, and does not serve client requests.

Step 4. Verify the status

Check the replication status on the secondary cluster:

d8 stronghold read -address="${SECONDARY_ADDR}" sys/replication/dr/status

stronghold read -address="${SECONDARY_ADDR}" sys/replication/dr/status

The status shows mode, state, connection_state, last_wal, and last_remote_wal. The DR secondary has caught up with the primary when its last_wal reaches last_remote_wal.

Preparing for a failover

A promote and other operations on a DR secondary (update-primary, disable) are authorized by the dr_operation_token parameter. Once the primary is unavailable, tokens can no longer be issued on it, so prepare them in advance.

In the dr_operation_token parameter, a DR secondary accepts the following tokens:

• A batch token issued on the primary. Its policy must allow update on the called endpoint, for example sys/replication/dr/secondary/promote.
• A DR operation token. It is issued on the DR secondary through a ceremony that requires a quorum of unseal-key shares (recovery-key shares for auto-unseal) and a one-time password (OTP).

The ceremony endpoints sys/replication/dr/secondary/generate-operation-token/* require authentication by default. On a DR secondary, the following tokens authenticate them: a batch token of the primary whose policy allows these endpoints, and the root token of the primary. A DR secondary does not accept service tokens of the primary other than the root token. The root token of the primary is not accepted as dr_operation_token.

The lifetime of a batch token is limited by the system maximum TTL (768 hours by default). A batch token cannot be renewed or revoked. Issue new tokens on a schedule before the current ones expire, and store them outside Stronghold, for example in a password manager.

Batch token for a promote

A batch token with the promote capability promotes a DR secondary without a ceremony and without key shares. Run the following commands on the primary: they create a policy, a role and a token.

d8 stronghold policy write dr-promote - <<'POLICY'
path "sys/replication/dr/secondary/promote" {
  capabilities = ["update"]
}
POLICY
d8 stronghold write auth/token/roles/dr-promote allowed_policies=dr-promote orphan=true token_type=batch renewable=false
d8 stronghold token create -role=dr-promote -ttl=168h

stronghold policy write dr-promote - <<'POLICY'
path "sys/replication/dr/secondary/promote" {
  capabilities = ["update"]
}
POLICY
stronghold write auth/token/roles/dr-promote allowed_policies=dr-promote orphan=true token_type=batch renewable=false
stronghold token create -role=dr-promote -ttl=168h

The token value in the output of the last command is a batch token with the hvb. prefix. It is valid for 168 hours.

Batch token for the ceremony

A batch token with capabilities on the ceremony endpoints lets a participant issue a DR operation token on the DR secondary. The ceremony still requires a quorum of key shares. Give such a token to each participant of the ceremony. Run the following commands on the primary:

d8 stronghold policy write dr-operation-token-participant - <<'POLICY'
path "sys/replication/dr/secondary/generate-operation-token/attempt" {
  capabilities = ["read", "update", "delete"]
}
path "sys/replication/dr/secondary/generate-operation-token/update" {
  capabilities = ["update"]
}
POLICY
d8 stronghold write auth/token/roles/dr-operation-token-participant allowed_policies=dr-operation-token-participant orphan=true token_type=batch renewable=false
d8 stronghold token create -role=dr-operation-token-participant -ttl=168h -display-name=<HOLDER_NAME>

stronghold policy write dr-operation-token-participant - <<'POLICY'
path "sys/replication/dr/secondary/generate-operation-token/attempt" {
  capabilities = ["read", "update", "delete"]
}
path "sys/replication/dr/secondary/generate-operation-token/update" {
  capabilities = ["update"]
}
POLICY
stronghold write auth/token/roles/dr-operation-token-participant allowed_policies=dr-operation-token-participant orphan=true token_type=batch renewable=false
stronghold token create -role=dr-operation-token-participant -ttl=168h -display-name=<HOLDER_NAME>

<HOLDER_NAME> is the name of the share holder. It becomes part of the token name and appears in the audit log.

Promote a DR secondary

The way to promote depends on the token you prepared.

Promote with a batch token

Pass the batch token for a promote in the dr_operation_token parameter:

curl --request POST --data '{"dr_operation_token":"<BATCH_TOKEN>"}' "${SECONDARY_ADDR}/v1/sys/replication/dr/secondary/promote"

In the command, <BATCH_TOKEN> is a batch token with the dr-promote policy.

After a promote, the former secondary becomes an active DR primary and serves clients.

Promote with a DR operation token

A DR operation token is issued on the DR secondary through a ceremony. Each participant runs the ceremony commands with their own token: a batch token for the ceremony or the root token of the primary.

1. Each participant sets their token in an environment variable:

   • Stronghold in DP
   • Stronghold in Linux

   export STRONGHOLD_TOKEN=<PARTICIPANT_TOKEN>

   export VAULT_TOKEN=<PARTICIPANT_TOKEN>

2. The initiator starts the ceremony. The command outputs a Nonce and an OTP. Save the OTP: it is required to decode the token:

   • Stronghold в DP
   • Stronghold в Linux

   d8 stronghold operator generate-root -address="${SECONDARY_ADDR}" -dr-token -init

   stronghold operator generate-root -address="${SECONDARY_ADDR}" -dr-token -init

3. Each share holder submits their share. The command prompts for the key share:

   • Stronghold в DP
   • Stronghold в Linux

   d8 stronghold operator generate-root -address="${SECONDARY_ADDR}" -dr-token -nonce=<NONCE>

   stronghold operator generate-root -address="${SECONDARY_ADDR}" -dr-token -nonce=<NONCE>

   After the last share, the command outputs an Encoded Token.

4. The initiator decodes the token. The command calls the DR secondary, so it also runs with the participant token:

   • Stronghold в DP
   • Stronghold в Linux

   d8 stronghold operator generate-root -address="${SECONDARY_ADDR}" -dr-token -decode=<ENCODED_TOKEN> -otp=<OTP>

   stronghold operator generate-root -address="${SECONDARY_ADDR}" -dr-token -decode=<ENCODED_TOKEN> -otp=<OTP>

5. Promote the secondary with the DR operation token:

   curl --request POST --data '{"dr_operation_token":"<DR_OPERATION_TOKEN>"}' "${SECONDARY_ADDR}/v1/sys/replication/dr/secondary/promote"

In the commands:

• <PARTICIPANT_TOKEN>: Token of the ceremony participant
• <NONCE>: Nonce value from the output of the -init command
• <ENCODED_TOKEN>: Encoded Token value after the last share
• <OTP>: OTP value from the output of the -init command
• <DR_OPERATION_TOKEN>: Decoded DR operation token

Promote without prepared tokens

If there are no prepared tokens and the primary is unavailable, temporarily allow the ceremony endpoints on the DR secondary without authentication. The ceremony still requires a quorum of key shares.

1. On every DR secondary node, add the enable_unauthenticated_access parameter to the server configuration:

   enable_unauthenticated_access = ["generate-operation-token"]

2. On every node, send the SIGHUP signal to the Stronghold process to reload the configuration. A restart is not required:

   sudo kill -s HUP $(pidof stronghold)

3. For the duration of the ceremony, restrict network access to the DR secondary API port (8200 by default) to the participants’ addresses, for example with firewall rules.

4. Run steps 2–5 from “Promote with a DR operation token” without the participant token variable.

5. Remove the enable_unauthenticated_access parameter from the configuration, send SIGHUP on every node again and lift the network restrictions.

Recover the former primary

After a failover, the cluster already has an active primary — the promoted secondary. Do not start the former primary back up as a primary, or the cluster ends up with two primaries. Instead, reconnect it to the new primary as a DR secondary.

1. On the new primary, create an activation token: d8 stronghold write sys/replication/dr/primary/secondary-token id=<SECONDARY_ID>. The wrap_info.token field of the response contains the activation token.
2. If the former primary still considers itself a primary, demote it: d8 stronghold write -force sys/replication/dr/primary/demote. After the demote, the former primary is a DR secondary.
3. On the former primary, generate a DR operation token. Run steps 1–4 from Promote with a DR operation token against the address of the former primary.
4. Reconnect the former primary to the new primary and pass the DR operation token: d8 stronghold write sys/replication/dr/secondary/update-primary token=<ACTIVATION_TOKEN> dr_operation_token=<DR_OPERATION_TOKEN>.

In the commands:

• <SECONDARY_ID>: Identifier of the DR secondary on the new primary
• <ACTIVATION_TOKEN>: Activation token from step 1
• <DR_OPERATION_TOKEN>: DR operation token from step 3

Use the token method: the new primary has a different identity, and primary_cluster_addr does not apply to it.

Management operations

The table lists the commands for managing DR replication:

Take the following into account:

• demote lowers a DR primary to a disabled DR secondary while keeping the cluster ID and local data, so it is ready to reconnect without a wipe.
• update-primary runs on a DR secondary and requires a DR operation token.
• Planned switchover: demote the current primary, promote the standby, then update-primary to bring the former primary back as a secondary. For details, refer to Recover the former primary.

Additional resources

• Regular disaster recovery drills
• Usage examples