Sizing

Secrets storage workloads vary a lot between organizations, so treat the values below as a starting point. Check actual resource usage with the metrics described in Monitoring and adjust the cluster size.

Baseline system requirements are listed in Requirements.

Cluster profiles

The small and large cluster values come from Requirements. The Medium profile and disk size are recommendations to validate against your workload.

Approximate performance depending on the number of cores (from Requirements):

Raft disk

Every write in Stronghold is committed to the Raft log on a majority of nodes with a disk sync, so disk latency directly affects write time and leader election stability.

Recommendations:

• use local SSD or NVMe disks; avoid network disks with unpredictable latency;
• use a dedicated partition for Raft data so that OS and audit logs cannot fill it up;
• monitor disk sync latency: aim for single-digit milliseconds at most;
• keep in mind that storage size grows with the number of leases and tokens, and Raft snapshots need space for temporary files.

You can check disk sync latency with fio:

fio --name=raft-fsync --directory=/opt/stronghold/data --rw=write --ioengine=sync --fdatasync=1 --bs=4k --size=100m

In the output, look at the fsync/fdatasync/sync_file_range percentiles. Remove the test file after the check.

Number of nodes

Recommendations:

• use an odd number of voting nodes: an even number does not increase fault tolerance;
• do not grow beyond five voting nodes: each additional node increases commit time;
• place nodes in different failure domains with low network latency between them;
• to scale reads, use performance standby (Stronghold EE); for disaster recovery, use DR replication.

In DP

In DP, the module sets the resources and the number of replicas as follows:

• Resource requests of the stronghold pod are 50m CPU and 256Mi memory; the VerticalPodAutoscaler (Initial mode) can raise them up to 500m and 1Gi. The stronghold-automatic pod requests 10m CPU and 50Mi memory.
• The default disk size (diskSizeGigabytes) is 1 GB; set a value sufficient for your data.
• On the initial installation, the number of replicas is calculated as the number of master nodes plus arbiter nodes; if the result is even, one more replica is added. After that, the current number of replicas of the StatefulSet is kept.
• The PodDisruptionBudget has minAvailable equal to replicas/2 + 1 (rounded down; 0 with a single replica).
• Destructive storage migrations require at least 3 replicas.

What drives load

Revisiting cluster size

Add resources if:

• the 99th percentile of stronghold_core_handle_request keeps growing;
• CPU usage on the active node regularly exceeds 70–80%;
• memory usage approaches the node RAM;
• stronghold_raft_commitTime grows and leader elections become more frequent.