KV1/KV2 replication

How KV1/KV2 replication works in Stronghold

Replication means automatic copying of secrets between several Stronghold instances in the master/slave mode using a pull model.

Replication is supported only for KV1 and KV2 stores.

Data is synchronized periodically on a schedule or according to the individual settings of a specific KV1/KV2 store.

For replication to work, you need to:

• Ensure network connectivity with the remote Stronghold cluster.
• Configure a correct TLS connection, if TLS is used.
• Obtain a token to access the remote cluster.
• Grant this token list and read permissions on the KV1/KV2 stores in the remote cluster.

To enable replication, set the replication parameters when mounting a new KV1/KV2 store.

Keep in mind:

• The remote and local mount paths may have different names.
• Replication can be configured between different namespaces in the local and remote clusters.
• You can configure replication of several local stores with different names from one remote store.
• If replication is configured for a local store, it works only in read-only mode.

Writing, modifying, and deleting secrets in a local replicated store is not possible. All changes must be made in the source master store.

After the next synchronization run, changes from the remote store are applied to the local one.

When replication is disabled, the read-only status is removed, and adding, modifying, and deleting secrets becomes available locally.

When replication is enabled again, all local changes are deleted or overwritten with data from the source store.

Configuring KV1/KV2 replication

Replication is configured on the consumer side, that is, on the slave Stronghold cluster, when mounting a new KV1/KV2 store.

The settings include the following parameters:

• Address of the remote Stronghold cluster
• Token to access the remote cluster
• wrapping token for secure transfer of the access token
• TLS certificate or path to the certificate for connecting to the remote cluster
• namespace path where the remote KV1/KV2 store is located
• mount path of the remote KV1/KV2 store
• List of secret path values to replicate
• Replication run period
• Enabling or disabling replication
• KV store version

Creating a replication token

The token for accessing the remote cluster must have list and read permissions on the replicated secrets.

If the issued token supports self-renewal, Stronghold automatically renews it for 30 days when the remaining TTL drops below 7 days and the maxTTL parameter is not exceeded.

Below is an example of creating a policy and a token for replication from the dev-secrets mount located in the ns_path_1 namespace:

d8 stronghold policy write -namespace=ns_path_1 replicate-dev-secrets - <<'EOF'
# Allow the token to list and read secrets in dev-secrets.
path "dev-secrets/*" {
  capabilities = ["read", "list"]
}

# Allow the token to read the dev-secrets mount settings.
path "sys/mounts/dev-secrets" {
  capabilities = ["read"]
}

# Allow the token to look up its own properties.
path "auth/token/lookup-self" {
  capabilities = ["read"]
}

# Allow the token to renew itself.
path "auth/token/renew-self" {
  capabilities = ["update"]
}
EOF

d8 stronghold token create -namespace=ns_path_1 -policy=replicate-dev-secrets -orphan=true -period=30d

stronghold policy write -namespace=ns_path_1 replicate-dev-secrets - <<'EOF'
# Allow the token to list and read secrets in dev-secrets.
path "dev-secrets/*" {
  capabilities = ["read", "list"]
}

# Allow the token to read the dev-secrets mount settings.
path "sys/mounts/dev-secrets" {
  capabilities = ["read"]
}

# Allow the token to look up its own properties.
path "auth/token/lookup-self" {
  capabilities = ["read"]
}

# Allow the token to renew itself.
path "auth/token/renew-self" {
  capabilities = ["update"]
}
EOF

stronghold token create -namespace=ns_path_1 -policy=replicate-dev-secrets -orphan=true -period=30d

Creating a wrapping token

A wrapping token lets you configure replication without passing the access token itself to the consumer cluster.

A wrapping token is a single-use token with a limited TTL that carries the real access token. When configuring replication, the wrapping token is passed to the consumer cluster, which then performs unwrap on the source cluster and obtains the real replication token.

Example of creating a wrapping token on the source cluster:

d8 stronghold token create -namespace=ns_path_1 -policy=replicate-dev-secrets -orphan=true -period=30d -wrap-ttl=5m -field=wrapping_token

stronghold token create -namespace=ns_path_1 -policy=replicate-dev-secrets -orphan=true -period=30d -wrap-ttl=5m -field=wrapping_token

Pass the resulting wrapping token when configuring replication on the consumer cluster.

Configuring replication using the CLI

Without TLS

To mount a replicated KV store that connects to the source cluster without TLS, run the following command on the consumer cluster:

d8 stronghold secrets enable -path=<LOCAL_MOUNT_PATH> -src-address=<SOURCE_ADDR> -src-wrapping-token=<WRAPPING_TOKEN> -src-namespace=<SOURCE_NAMESPACE> -src-mount-path=<SOURCE_MOUNT_PATH> -sync-period-min=<SYNC_PERIOD_MIN> -version=<KV_VERSION> -namespace=<LOCAL_NAMESPACE> kv

stronghold secrets enable -path=<LOCAL_MOUNT_PATH> -src-address=<SOURCE_ADDR> -src-wrapping-token=<WRAPPING_TOKEN> -src-namespace=<SOURCE_NAMESPACE> -src-mount-path=<SOURCE_MOUNT_PATH> -sync-period-min=<SYNC_PERIOD_MIN> -version=<KV_VERSION> -namespace=<LOCAL_NAMESPACE> kv

With TLS

To mount a replicated KV store that connects to the source cluster over TLS, pass the CA certificate of the source cluster in -src-ca-cert:

d8 stronghold secrets enable -path=<LOCAL_MOUNT_PATH> -src-address=<SOURCE_ADDR> -src-wrapping-token=<WRAPPING_TOKEN> -src-namespace=<SOURCE_NAMESPACE> -src-mount-path=<SOURCE_MOUNT_PATH> -src-ca-cert=@<CA_CERT_FILE> -sync-period-min=<SYNC_PERIOD_MIN> -version=<KV_VERSION> -namespace=<LOCAL_NAMESPACE> kv

stronghold secrets enable -path=<LOCAL_MOUNT_PATH> -src-address=<SOURCE_ADDR> -src-wrapping-token=<WRAPPING_TOKEN> -src-namespace=<SOURCE_NAMESPACE> -src-mount-path=<SOURCE_MOUNT_PATH> -src-ca-cert=@<CA_CERT_FILE> -sync-period-min=<SYNC_PERIOD_MIN> -version=<KV_VERSION> -namespace=<LOCAL_NAMESPACE> kv

Replace the placeholders with the values of the corresponding parameters:

• -path: Name of the mount path of the local KV store where the data is copied.
• -src-address: Address of the remote Stronghold cluster.
• -src-token: Token to access the remote cluster. Required if -src-wrapping-token is not specified.
• -src-wrapping-token: Wrapping token to be unwrapped on the remote cluster. Required if -src-token is not specified.
• -src-namespace: namespace path in the remote cluster. The default is root.
• -src-mount-path: Name of the mount path of the remote KV store.
• -src-secret-path: List of secret path values to replicate.
• -src-ca-cert: CA certificate for the TLS connection. If the certificate is in a file, use the @ca-cert.pem form.
• -sync-period-min: Synchronization period in minutes. The default is 1.
• -version: KV store version: 1 or 2.
• -namespace: namespace path in the local cluster. The default is root.

Changing replication settings using the CLI

You can edit the following:

• Token for accessing the remote cluster
• Wrapping token for replacing the replication credentials
• TLS certificate
• List of secret path values
• Replication run period
• Enabling and disabling replication

Example of changing the settings. <SECRET_PATHS> is a list of secret path values in the source cluster:

d8 stronghold secrets tune -src-wrapping-token=<WRAPPING_TOKEN> -src-secret-path=<SECRET_PATHS> -src-ca-cert=@<CA_CERT_FILE> -sync-enable=true -sync-period-min=<SYNC_PERIOD_MIN> -namespace=<LOCAL_NAMESPACE> <LOCAL_MOUNT_PATH>

stronghold secrets tune -src-wrapping-token=<WRAPPING_TOKEN> -src-secret-path=<SECRET_PATHS> -src-ca-cert=@<CA_CERT_FILE> -sync-enable=true -sync-period-min=<SYNC_PERIOD_MIN> -namespace=<LOCAL_NAMESPACE> <LOCAL_MOUNT_PATH>

To disable replication:

d8 stronghold secrets tune -sync-enable=false -namespace=<LOCAL_NAMESPACE> <LOCAL_MOUNT_PATH>

stronghold secrets tune -sync-enable=false -namespace=<LOCAL_NAMESPACE> <LOCAL_MOUNT_PATH>

To enable it again:

d8 stronghold secrets tune -sync-enable=true -namespace=<LOCAL_NAMESPACE> <LOCAL_MOUNT_PATH>

stronghold secrets tune -sync-enable=true -namespace=<LOCAL_NAMESPACE> <LOCAL_MOUNT_PATH>

To read the current settings:

d8 stronghold read -namespace=<LOCAL_NAMESPACE> sys/mounts/<LOCAL_MOUNT_PATH>/tune

stronghold read -namespace=<LOCAL_NAMESPACE> sys/mounts/<LOCAL_MOUNT_PATH>/tune

Additional resources

• Migrating from HashiCorp Vault
• Usage examples