Delivering secrets to Kubernetes pods

An application in Kubernetes can get secrets from Stronghold in several ways. All of them rely on the Kubernetes auth method: a pod presents its ServiceAccount token, and Stronghold issues a token with the policies bound to the role.

Comparison

The table compares the methods by where the secret is stored and how it is updated:

Consider the following when choosing a method:

• If a secret must not be stored in etcd, choose CSI, env-injector, Stronghold Agent, or direct API calls.
• Secrets in environment variables are visible to processes that can read /proc/<PID>/environ and may end up in dumps and logs. Files in a volume are not exposed this way.
• For dynamic secrets and short-lived certificates, use Stronghold Agent as a sidecar or direct API calls: only these renew leases during the pod lifetime.

Preparation: role and policy

The example below applies to all methods. It assumes that the KV version 2 secrets engine is enabled at secret, and the Kubernetes auth method at kubernetes.

1. Create a policy that allows reading the application secrets:

   d8 stronghold policy write myapp-read - <<'POLICY'
   path "secret/data/myapp/*" {
     capabilities = ["read"]
   }
   POLICY

2. Store the secret:

   d8 stronghold kv put -mount=secret myapp/db username=app password=S3cr3t

3. Create a role that binds the myapp ServiceAccount in the myapp namespace to the policy:

   d8 stronghold write auth/kubernetes/role/myapp bound_service_account_names=myapp bound_service_account_namespaces=myapp token_policies=myapp-read token_ttl=1h

   In DP in Automatic mode, the Kubernetes auth method for the current cluster is created by the module at the kubernetes_local path: in this case, use auth/kubernetes_local instead of auth/kubernetes here and in the examples below (mount_path, login URL). The auth/kubernetes path applies to a manually enabled method.

4. Create the ServiceAccount in the cluster:

   d8 k create namespace myapp
   d8 k -n myapp create serviceaccount myapp

Configuring the auth method itself (auth/kubernetes/config) is described in Kubernetes auth method. In Deckhouse Platform, the stronghold module in Automatic mode connects the current cluster automatically for the secrets-store-integration module.

secrets-store-integration CSI driver

The Deckhouse Platform module secrets-store-integration mounts secrets into the pod as files using a CSI driver. The secret is not stored in a Secret object and does not reach etcd.

Example of a SecretsStoreImport and a pod that mounts it:

apiVersion: deckhouse.io/v1alpha1
kind: SecretsStoreImport
metadata:
  name: myapp-db
  namespace: myapp
spec:
  type: CSI
  role: myapp
  files:
    - name: db-password
      source:
        path: secret/data/myapp/db
        key: password
---
apiVersion: v1
kind: Pod
metadata:
  name: myapp
  namespace: myapp
spec:
  serviceAccountName: myapp
  containers:
    - name: app
      image: registry.example.com/myapp:1.0.0
      volumeMounts:
        - name: secrets
          mountPath: /mnt/secrets
          readOnly: true
  volumes:
    - name: secrets
      csi:
        driver: secrets-store.csi.deckhouse.io
        volumeAttributes:
          secretsStoreImport: myapp-db

The application reads the password from /mnt/secrets/db-password. The CSI driver updates the file automatically every two minutes. To use a new secret value, the application must re-read the file.

secrets-store-integration env-injector

The env-injector from the same module substitutes secret values into environment variables when the container starts. The values exist only in process memory and are not written to the pod spec.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  namespace: myapp
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
      annotations:
        secrets-store.deckhouse.io/role: myapp
    spec:
      serviceAccountName: myapp
      containers:
        - name: app
          image: registry.example.com/myapp:1.0.0
          env:
            - name: DB_USER
              value: secrets-store:secret/data/myapp/db#username
            - name: DB_PASSWORD
              value: secrets-store:secret/data/myapp/db#password

To make the application pick up a new secret value, restart the pod, for example with d8 k -n myapp rollout restart deployment myapp.

Stronghold Agent

Stronghold Agent authenticates using the Kubernetes method, fetches secrets, and renders them to files using templates. In a pod, Agent runs:

• as an init container with exit_after_auth = true — secrets are rendered once at pod start;
• as a sidecar container — Agent renews the token and leases and re-renders templates when secrets change.

Files are shared with the application through an emptyDir volume with medium: Memory, so secrets are not written to the node disk.

Example Agent configuration in a ConfigMap (the section format is described in Settings):

apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-agent
  namespace: myapp
data:
  agent.hcl: |
    stronghold {
      address = "https://stronghold.example.com"
    }

    auto_auth {
      method "kubernetes" {
        mount_path = "auth/kubernetes"
        config = {
          role = "myapp"
        }
      }
    }

    template {
      destination = "/secrets/db.env"
      contents    = <<-EOT
      {{ with secret "secret/data/myapp/db" }}
      DB_USER={{ .Data.data.username }}
      DB_PASSWORD={{ .Data.data.password }}
      {{ end }}
      EOT
    }

The pod example below uses the image of the stronghold module from the Deckhouse container registry. The entrypoint of this image is stronghold, so args contains agent and -config= with the path to agent.hcl. In <STRONGHOLD_IMAGE>, specify the address of this image.

An image built with stronghold bootstrap docker has no ENTRYPOINT (for details, see Local Stronghold for development). For such an image, set command: ["stronghold", "agent", "-config=/etc/stronghold-agent/agent.hcl"] instead of args.

Example pod with Agent as a sidecar:

apiVersion: v1
kind: Pod
metadata:
  name: myapp
  namespace: myapp
spec:
  serviceAccountName: myapp
  containers:
    - name: stronghold-agent
      image: <STRONGHOLD_IMAGE>
      args: ["agent", "-config=/etc/stronghold-agent/agent.hcl"]
      volumeMounts:
        - name: agent-config
          mountPath: /etc/stronghold-agent
        - name: secrets
          mountPath: /secrets
    - name: app
      image: registry.example.com/myapp:1.0.0
      volumeMounts:
        - name: secrets
          mountPath: /secrets
          readOnly: true
  volumes:
    - name: agent-config
      configMap:
        name: myapp-agent
    - name: secrets
      emptyDir:
        medium: Memory

For the init container variant, move the stronghold-agent container to initContainers and add exit_after_auth = true to agent.hcl.

External Secrets Operator

External Secrets Operator is a third-party operator with a vault provider. Stronghold is API-compatible with Vault, so the provider can be pointed at Stronghold.

ESO creates a regular Secret object, so the secret value is stored in etcd. Enable encryption of Secret objects in etcd using cluster tooling and restrict access to Secrets with RBAC.

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: stronghold
  namespace: myapp
spec:
  provider:
    vault:
      server: https://stronghold.example.com
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: myapp
          serviceAccountRef:
            name: myapp
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: myapp-db
  namespace: myapp
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: stronghold
    kind: SecretStore
  target:
    name: myapp-db
  data:
    - secretKey: password
      remoteRef:
        key: myapp/db
        property: password

Direct API call

The application can authenticate with its ServiceAccount token and read the secret on its own. Example with curl:

SA_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

STRONGHOLD_TOKEN=$(curl -s --request POST --data "{\"role\": \"myapp\", \"jwt\": \"${SA_TOKEN}\"}" https://stronghold.example.com/v1/auth/kubernetes/login | jq -r '.auth.client_token')

curl -s --header "X-Vault-Token: ${STRONGHOLD_TOKEN}" https://stronghold.example.com/v1/secret/data/myapp/db | jq '.data.data'

SDK examples and guidance on renewing tokens and leases are in Application clients.