Upgrade

A Stronghold HA cluster with Raft storage is upgraded one node at a time. Standby nodes are upgraded first; then the active node hands over its role to one of the upgraded nodes and is upgraded last. This order keeps the quorum and the cluster available during the upgrade.

Preparation

Prepare the cluster and the new binary before the upgrade:

1. Read the release notes for all versions between the current and target ones. Pay attention to configuration changes and breaking changes.

2. Check the cluster version and state:

   d8 stronghold status
   d8 stronghold operator raft list-peers
   d8 stronghold operator raft autopilot state

   All nodes must be voter, and Autopilot must report Healthy: true.

3. Take a storage snapshot and inspect it (see Creating a snapshot and Inspecting a snapshot):

   d8 stronghold operator raft snapshot save pre-upgrade.snap
   d8 stronghold operator raft snapshot inspect pre-upgrade.snap

4. Back up the configuration file, systemd unit, and current binary on each node:

   cp -a /opt/stronghold/stronghold /opt/stronghold/stronghold.bak
   cp -a /opt/stronghold/config.hcl /opt/stronghold/config.hcl.bak
   cp -a /etc/systemd/system/stronghold.service /etc/systemd/system/stronghold.service.bak

5. If Shamir seal is used, make sure unseal key holders are available: every node must be unsealed after a restart.

6. Copy the new binary to all nodes, for example to /tmp/stronghold, and check its version:

   /tmp/stronghold version

7. Identify the active node: in the d8 stronghold operator raft list-peers output, its state is leader.

Upgrading standby nodes

Perform the steps on each standby node in turn. Do not move to the next node until the current one has rejoined the cluster.

1. Replace the binary:

   install -o root -g root -m 0755 /tmp/stronghold /opt/stronghold/stronghold

2. Restart the service:

   systemctl restart stronghold

3. If Shamir seal is used, unseal the node. Point to the node being upgraded:

   export STRONGHOLD_ADDR=https://raft-node-2.demo.tld:8200
   d8 stronghold operator unseal

4. Check that the node has rejoined the cluster and is unsealed:

   d8 stronghold status
   d8 stronghold operator raft autopilot state

   Wait until Autopilot shows the node as healthy and the cluster as Healthy: true.

5. Check the node server logs for errors:

   journalctl -u stronghold.service --since "10 minutes ago"

Upgrading the active node

Upgrade the active node after all standby nodes are upgraded:

1. Hand over the active role to one of the upgraded nodes. Run the command against the current active node:

   export STRONGHOLD_ADDR=https://raft-node-1.demo.tld:8200
   d8 stronghold operator step-down

2. Make sure another node has become the leader:

   d8 stronghold operator raft list-peers

3. Upgrade the former active node the same way as standby nodes: replace the binary, restart the service, unseal the node, and wait until it rejoins the cluster.

Post-upgrade checks

After all nodes are upgraded, check the cluster:

1. Check the version on each node:

   for node in raft-node-1 raft-node-2 raft-node-3; do
     STRONGHOLD_ADDR=https://${node}.demo.tld:8200 d8 stronghold status | grep -E 'Version|HA Mode|Sealed'
   done

2. Check the cluster state:

   d8 stronghold operator raft list-peers
   d8 stronghold operator raft autopilot state

3. Check the main scenarios: login via the auth methods in use, reading and writing a test secret, issuing dynamic credentials.

4. Check that metrics and alerts work (see Monitoring).

The autopilot state output shows each node version (Version), which lets you confirm that all nodes are upgraded.

Rollback

If a node fails to start after the upgrade or the cluster misbehaves:

1. Stop the upgraded node and restore the previous binary from /opt/stronghold/stronghold.bak.
2. If the new version has already modified the storage data and the previous version does not start, restore the cluster from the pre-upgrade.snap snapshot on the previous version (see Restoring from a snapshot).