The secrets-store-integration module: examples

This section contains usage examples for the secrets-store-integration module.

CLI tool d8 for Stronghold commands

Deckhouse CLI (d8) is a universal tool required to run commands such as d8 stronghold in the terminal.

To install d8, use one of the methods described in the CLI tool documentation.

Configuring the module to work with Deckhouse Stronghold

1. Enable the stronghold module by following the instructions.

2. To enable the secrets-store-integration module, apply the following resource:

   apiVersion: deckhouse.io/v1alpha1
   kind: ModuleConfig
   metadata:
     name: secrets-store-integration
   spec:
     enabled: true
     version: 1

   You do not have to set the connectionConfiguration parameter, because DiscoverLocalStronghold is used by default.

Configuring the module to work with an external store

The module requires a preconfigured secret store compatible with HashiCorp Vault. An authentication path must already be configured in the store. An example of store configuration is shown below.

To ensure that each API request is encrypted, sent, and processed by the correct recipient, you need a valid public Certificate Authority certificate used by the secret store. This CA public certificate in PEM format must be used as the caCert variable in the module configuration.

Example module configuration for using a Vault-compatible secret store running at secretstoreexample.com on the default TLS port (443):

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: secrets-store-integration
spec:
  version: 1
  enabled: true
  settings:
    connection:
      url: "https://secretstoreexample.com"
      authPath: "main-kube"
      caCert: |
        -----BEGIN CERTIFICATE-----
        MIIFoTCCA4mgAwIBAgIUX9kFz7OxlBlALMEj8WsegZloXTowDQYJKoZIhvcNAQEL
        ................................................................
        WoR9b11eYfyrnKCYoSqBoi2dwkCkV1a0GN9vStwiBnKnAmV3B8B5yMnSjmp+42gt
        o2SYzqM=
        -----END CERTIFICATE-----
    connectionConfiguration: Manual

Preparing a test environment

export VAULT_TOKEN=xxxxxxxxxxx
export VAULT_ADDR=https://secretstoreexample.com

Before injecting secrets, prepare a test environment.

1. Create a kv2 secret in Stronghold at demo-kv/myapp-secret and put the DB_USER and DB_PASS values there.

   • Enable and create the Key-Value store:

     d8 stronghold secrets enable -path=demo-kv -version=2 kv

     Alternative using curl:

     curl \
       --header "X-Vault-Token: ${VAULT_TOKEN}" \
       --request POST \
       --data '{"type":"kv","options":{"version":"2"}}' \
       ${VAULT_ADDR}/v1/sys/mounts/demo-kv

   • Set the database username and password as the secret value:

     d8 stronghold kv put demo-kv/myapp-secret DB_USER="username" DB_PASS="secret-password"

     Alternative using curl:

     curl \
       --header "X-Vault-Token: ${VAULT_TOKEN}" \
       --request PUT \
       --data '{"data":{"DB_USER":"username","DB_PASS":"secret-password"}}' \
       ${VAULT_ADDR}/v1/demo-kv/data/myapp-secret

   • Verify the stored secret:

     d8 stronghold kv get demo-kv/myapp-secret

     Alternative verification using curl:

     curl \
       --header "X-Vault-Token: ${VAULT_TOKEN}" \
       ${VAULT_ADDR}/v1/demo-kv/data/myapp-secret

2. If necessary, add an authentication path (authPath) for authentication and authorization in Stronghold using the Kubernetes API of a remote cluster.

   • By default, Stronghold enables and configures the Kubernetes authentication method under the name kubernetes_local for the cluster where Stronghold itself is running. If you need to configure access through remote clusters, set the authentication path (authPath) and enable authentication and authorization in Stronghold through the Kubernetes API for each cluster:

     d8 stronghold auth enable -path=remote-kube-1 kubernetes

     Alternative using curl:

     curl \
       --header "X-Vault-Token: ${VAULT_TOKEN}" \
       --request POST \
       --data '{"type":"kubernetes"}' \
       ${VAULT_ADDR}/v1/sys/auth/remote-kube-1

   • Set the Kubernetes API address for each cluster:

     d8 stronghold write auth/remote-kube-1/config \
       kubernetes_host="https://api.kube.my-deckhouse.com"

     Alternative using curl:

     curl \
       --header "X-Vault-Token: ${VAULT_TOKEN}" \
       --request PUT \
       --data '{"kubernetes_host":"https://api.kube.my-deckhouse.com"}' \
       ${VAULT_ADDR}/v1/auth/remote-kube-1/config

3. Create a myapp-ro-policy policy in Stronghold that allows reading secrets from demo-kv/data/myapp-secret:

   d8 stronghold policy write myapp-ro-policy - <<EOF
   path "demo-kv/data/myapp-secret" {
     capabilities = ["read"]
   }
   EOF

   Alternative using curl:

   curl \
     --header "X-Vault-Token: ${VAULT_TOKEN}" \
     --request PUT \
     --data '{"policy":"path \"demo-kv/data/myapp-secret\" {\n capabilities = [\"read\"]\n}\n"}' \
     ${VAULT_ADDR}/v1/sys/policies/acl/myapp-ro-policy

4. Create a role in Stronghold for the myapp-sa service account in the myapp-namespace namespace and bind the policy created earlier to it.

   In addition to the Stronghold-side configuration, you must configure authorization permissions for the ServiceAccount objects used in the Kubernetes cluster.

   See the required settings in the next section.

   • Create a role consisting of the namespace and policy name. Bind it to the myapp-sa ServiceAccount in the myapp-namespace namespace and to the myapp-ro-policy policy:

     The recommended TTL value for the Kubernetes token is 10m.

     d8 stronghold write auth/kubernetes_local/role/myapp-role \
         bound_service_account_names=myapp-sa \
         bound_service_account_namespaces=myapp-namespace \
         policies=myapp-ro-policy \
         ttl=10m

     Alternative using curl:

     curl \
       --header "X-Vault-Token: ${VAULT_TOKEN}" \
       --request PUT \
       --data '{"bound_service_account_names":"myapp-sa","bound_service_account_namespaces":"myapp-namespace","policies":"myapp-ro-policy","ttl":"10m"}' \
       ${VAULT_ADDR}/v1/auth/kubernetes_local/role/myapp-role

   • Repeat the same for remote clusters, specifying a different authentication path:

     d8 stronghold write auth/remote-kube-1/role/myapp-role \
         bound_service_account_names=myapp-sa \
         bound_service_account_namespaces=myapp-namespace \
         policies=myapp-ro-policy \
         ttl=10m

     Alternative using curl:

     curl \
       --header "X-Vault-Token: ${VAULT_TOKEN}" \
       --request PUT \
       --data '{"bound_service_account_names":"myapp-sa","bound_service_account_namespaces":"myapp-namespace","policies":"myapp-ro-policy","ttl":"10m"}' \
       ${VAULT_ADDR}/v1/auth/remote-kube-1/role/myapp-role

   These settings allow any pod in the myapp-namespace namespace in both Kubernetes clusters that uses the myapp-sa ServiceAccount to authenticate and authorize in Stronghold to read secrets according to the myapp-ro-policy policy.

5. Create the myapp-namespace namespace in the cluster:

   d8 k create namespace myapp-namespace

6. Create the myapp-sa service account in that namespace:

   d8 k -n myapp-namespace create serviceaccount myapp-sa

How to allow a ServiceAccount to authenticate in Stronghold

To authenticate in Stronghold, a pod uses the token generated for its ServiceAccount. For Stronghold to validate the provided ServiceAccount data, the Stronghold service must have get, list, and watch permissions for the tokenreviews.authentication.k8s.io and subjectaccessreviews.authorization.k8s.io endpoints. You can also use the system:auth-delegator ClusterRole for this.

Stronghold can use different credentials to send requests to the Kubernetes API:

• A token of the application that is trying to authenticate in Stronghold. In this case, every service authenticating in Stronghold requires the system:auth-delegator ClusterRole or the API permissions listed above on the ServiceAccount it uses. See the example in the Stronghold documentation.
• A static token of a ServiceAccount created specifically for Stronghold and granted the necessary permissions. Configuring Stronghold for this case is described in detail in the Stronghold documentation.

Injecting environment variables

How injection works

When the module is enabled, a mutating-webhook appears in the cluster. If a pod has the secrets-store.deckhouse.io/role annotation, the webhook modifies the pod manifest by adding the injector.

In the modified pod:

1. An init container is added.
2. The init container copies a statically linked injector binary from the service image into a temporary directory shared by all containers in the pod.
3. In the remaining containers, the original startup commands are replaced with a command that launches the injector binary.
4. The injector retrieves the required data from a Vault-compatible store using the application’s service account.
5. It puts these variables into the process ENV.
6. It performs the execve system call and starts the original command.

If a container does not define a startup command in the pod manifest, the image manifest is fetched from the registry and the command is taken from it.

Credentials from imagePullSecrets specified in the pod manifest are used to retrieve the manifest from a private image registry.

Injector annotations

The following annotations are available to modify injector behavior:

Using the injector, you can specify templates in pod manifests instead of actual env values. They are replaced with values from the store at container startup time.

env:
  - name: MY_SECRET
    value: secrets-store:demo-kv/data/myapp-secret#password

Example of retrieving the DB_PASS key from a kv2 secret at demo-kv/myapp-secret from a Vault-compatible store:

env:
  - name: PASSWORD
    value: secrets-store:demo-kv/data/myapp-secret#DB_PASS

Example of retrieving version 4 of the DB_PASS key from a kv2 secret at demo-kv/myapp-secret:

env:
  - name: PASSWORD
    value: secrets-store:demo-kv/data/myapp-secret#DB_PASS#4

The template can also be stored in a ConfigMap or a Secret and connected using envFrom:

envFrom:
  - secretRef:
      name: app-secret-env
  - configMapRef:
      name: app-env

Actual secrets from the Vault-compatible store are injected only at application startup. The Secret and ConfigMap objects contain templates.

Importing variables from a store path

In this scenario, all keys from a single secret are imported.

1. Create a pod named myapp1 that imports all variables from the store at demo-kv/data/myapp-secret:

   kind: Pod
   apiVersion: v1
   metadata:
     name: myapp1
     namespace: myapp-namespace
     annotations:
       secrets-store.deckhouse.io/role: "myapp-role"
       secrets-store.deckhouse.io/env-from-path: demo-kv/data/common-secret,demo-kv/data/myapp-secret
   spec:
     serviceAccountName: myapp-sa
     containers:
     - image: alpine:3.20
       name: myapp
       command:
       - sh
       - -c
       - while printenv; do sleep 5; done

2. Apply the manifest:

   d8 k create --filename myapp1.yaml

3. Check the pod logs after startup. The output should include all variables from demo-kv/data/myapp-secret:

   d8 k -n myapp-namespace logs myapp1

4. Delete the pod:

   d8 k -n myapp-namespace delete pod myapp1 --force

Importing explicitly defined variables from the store

1. Create a test pod named myapp2 that imports the required variables from the store using templates:

   kind: Pod
   apiVersion: v1
   metadata:
     name: myapp2
     namespace: myapp-namespace
     annotations:
       secrets-store.deckhouse.io/role: "myapp-role"
   spec:
     serviceAccountName: myapp-sa
     containers:
     - image: alpine:3.20
       env:
       - name: DB_USER
         value: secrets-store:demo-kv/data/myapp-secret#DB_USER
       - name: DB_PASS
         value: secrets-store:demo-kv/data/myapp-secret#DB_PASS
       name: myapp
       command:
       - sh
       - -c
       - while printenv; do sleep 5; done

2. Apply the configuration:

   d8 k create --filename myapp2.yaml

3. Check the pod logs after startup. The output should include variables from demo-kv/data/myapp-secret:

   d8 k -n myapp-namespace logs myapp2

4. Delete the pod:

   d8 k -n myapp-namespace delete pod myapp2 --force

Mounting a secret from the store as a file into a container

Use the SecretsStoreImport custom resource to deliver secrets to the application.

This example uses the myapp-sa service account and the myapp-namespace namespace created during test environment preparation.

1. Create a SecretsStoreImport custom resource named myapp-ssi in the cluster:

   apiVersion: deckhouse.io/v1alpha1
   kind: SecretsStoreImport
   metadata:
     name: myapp-ssi
     namespace: myapp-namespace
   spec:
     type: CSI
     role: myapp-role
     files:
       - name: "db-password"
         source:
           path: "demo-kv/data/myapp-secret"
           key: "DB_PASS"

2. Create a test pod named myapp3 in the cluster that mounts the secret from the store as a file:

   kind: Pod
   apiVersion: v1
   metadata:
     name: myapp3
     namespace: myapp-namespace
   spec:
     serviceAccountName: myapp-sa
     containers:
     - image: alpine:3.20
       name: backend
       command:
       - sh
       - -c
       - while cat /mnt/secrets/db-password; do echo; sleep 5; done
       volumeMounts:
       - name: secrets
         mountPath: "/mnt/secrets"
     volumes:
     - name: secrets
       csi:
         driver: secrets-store.csi.deckhouse.io
         volumeAttributes:
           secretsStoreImport: "myapp-ssi"

   After these resources are applied, a pod is created with a backend container. Inside the container filesystem, the /mnt/secrets directory contains the mounted secrets volume. This directory contains the db-password file with the database password (DB_PASS) from the Stronghold Key-Value store.

3. Check the pod logs after startup. The output should contain the contents of /mnt/secrets/db-password:

   d8 k -n myapp-namespace logs myapp3

4. Delete the pod:

   d8 k -n myapp-namespace delete pod myapp3 --force

Delivering binary files into a container

In some cases, you may need to deliver a binary file into a container, for example:

• A JKS keystore
• A keytab for Kerberos authentication

In this case, you can encode the binary file as Base64 and place it into the secret store. When retrieved, the CSI driver decodes the data and places the binary file into the container. To do this, set decodeBase64 to true for the corresponding file.

If decoding fails, for example because the store contains invalid Base64 data, the container will not be created.

Example:

1. Encode the file as Base64 and place it into the store:

   d8 stronghold kv put demo-kv/myapp-secret keytab=$(cat /path/to/keytab_file | base64 -w0)

2. Create a SecretsStoreImport manifest with the decoding parameter set:

   apiVersion: deckhouse.io/v1alpha1
   kind: SecretsStoreImport
   metadata:
     name: myapp-ssi
     namespace: myapp-namespace
   spec:
     type: CSI
     role: myapp-role
     files:
       - name: "keytab"
         decodeBase64: true
         source:
           path: "demo-kv/data/myapp-secret"
           key: "keytab"

3. A binary file named keytab will be created in the container.

Autorotation feature

The autorotation feature in the secrets-store-integration module is enabled by default. Every two minutes, the module polls Stronghold and synchronizes secrets in the mounted file if they have changed.

There are two ways to track changes to the secret file in a pod:

• watch the modification time of the mounted file and react when it changes;
• use the inotify API, which provides a file system event subscription mechanism.

Inotify is part of the Linux kernel. Once a change is detected, there are many possible responses depending on the application architecture and programming language. The simplest option is to make Kubernetes restart the pod by failing the livenessProbe.

Example of using inotify in a Python application:

#!/usr/bin/python3
import inotify.adapters

def _main():
    i = inotify.adapters.Inotify()
    i.add_watch('/mnt/secrets-store/db-password')
    for event in i.event_gen(yield_nones=False):
        (_, type_names, path, filename) = event
        if 'IN_MODIFY' in type_names:
            print("file modified")

if __name__ == '__main__':
    _main()

Example of using inotify in a Go application:

watcher, err := inotify.NewWatcher()
if err != nil {
    log.Fatal(err)
}

err = watcher.Watch("/mnt/secrets-store/db-password")
if err != nil {
    log.Fatal(err)
}

for {
    select {
    case ev := <-watcher.Event:
        if ev == 'InModify' {
            log.Println("file modified")
        }
    case err := <-watcher.Error:
        log.Println("error:", err)
    }
}

Limitations when updating secrets

Files with secrets are not updated if subPath is used.

volumeMounts:
- mountPath: /app/settings.ini
  name: app-config
  subPath: settings.ini
...
volumes:
- name: app-config
  csi:
    driver: secrets-store.csi.deckhouse.io
    volumeAttributes:
      secretsStoreImport: "python-backend"