The Stronghold module module: usage

Enabling the module

You can enable the module in one of the following ways:

• Using the Deckhouse web interface.

  In the “System” section → “System Management” → “Deckhouse” → “Modules”, open the stronghold module, enable the “Module enabled” switch. Save the changes.

• Using Deckhouse CLI.

  Execute the following command to enable the module:

  d8 system module enable stronghold
  

• Using ModuleConfig stronghold.

  Set spec.enabled to true or false in ModuleConfig stronghold (create it if necessary).

  Example manifest to enable the module:

  apiVersion: deckhouse.io/v1alpha1
  kind: ModuleConfig
  metadata:
    name: stronghold
  spec:
    enabled: true
  

By default, the module will run in the Automatic mode with the Ingress inlet. In the current version, there are no other inlets and modes.

Possible Errors when enabling the module

secret “ingress-tls” not found

Check the status of the stronghold pod:

d8 k -n d8-stronghold describe pod stronghold-0

The output contains the string:

MountVolume.SetUp failed for volume "certificates" : secret "ingress-tls" not found

If automatic TLS certificate ordering is used, the reason might be that the TLS certificate is not yet ready.

It may take some time (usually up to 10 minutes) for the certificate to be ordered and for the ingress-tls secret to appear in the d8-stronghold namespace. After the certificate is successfully ordered, Stronghold might not start immediately due to exponential backoff for restarting containers in Kubernetes. In this case, you can either wait for it to start (the time depends on how long Stronghold couldn’t start), or restart Stronghold manually by executing the command:

d8 k -n d8-stronghold delete po -l app.kubernetes.io/instance=stronghold

To diagnose the issue with ordering a TLS certificate, follow these steps:

• Check the status of the TLS certificate order using the command:

  d8 k -n d8-stronghold get certificate
  

  If the certificate is obtained (status Ready), you can either wait for Stronghold to launch or restart it manually (see above).

• Get the list of CertificateRequest:

  d8 k -n d8-stronghold get certificaterequest
  

  If such objects exist, find among them the object starting with stronghold- (in the example, stronghold-b5wc6 will be used).

• Check the status of the CertificateRequest object:

  d8 k -n d8-stronghold describe certificaterequest stronghold-b5wc6
  

  One of the reasons might be the error too many certificates already issued for <DOMAIN>. This often occurs due to exceeding the request limit if a free service is used (such as sslip.io or getmoss.site). In this case, you either need to wait for the timeout limit to pass or change the method of certificate creation for the domain stronghold.* (for example, choose a different ClusterIssuer, use a self-signed certificate, or manually sign the certificate).

  Upon successful receipt of the certificate, the CertificateRequest status should contain the following lines:

  Message:               Certificate fetched from issuer successfully
  Reason:                Issued
  Status:                True
  Type:                  Ready
  

  Also, the d8-stronghold namespace must contain a secret (Secret) named ingress-tls with the type kubernetes.io/tls.

The stronghold cannot start due to a Readiness Probe error

Run the command:

d8 k -n d8-stronghold get po

Example output:

$ d8 k -n d8-stronghold get po
NAME                                   READY   STATUS                       RESTARTS   AGE
stronghold-0                           1/2     Running                      0          33s
stronghold-automatic-594b78ff6-b4pgj   0/1     CreateContainerConfigError   0          15s

If stronghold cannot start due to a Readiness Probe error (example above), check the status with the following command:

d8 k -n d8-stronghold exec stronghold-0 -it -- stronghold status

Example output:

$ d8 k -n d8-stronghold exec stronghold-0 -it -- stronghold status
Defaulted container "stronghold" out of: stronghold, kube-rbac-proxy
Key                Value
---                -----
...
Initialized        true
Sealed             true
...
Unseal Progress    0/1   <-
Unseal Nonce       n/a   <-
...

This message means that Stronghold was previously launched on the node but was removed. The data in the storage was preserved, but Stronghold cannot use it.

Follow these steps to delete old Stronghold data if you do not plan to use them:

1. Reduce the number of Stronghold replicas to 0:

   d8 k -n d8-stronghold scale statefulset stronghold --replicas=0
   

2. Run the command on all master nodes of the cluster to remove old Stronghold data:

   sudo rm -Ir /var/lib/deckhouse/stronghold/{node-id,vault.db,raft}
   

3. Increase the number of Stronghold replicas back to the required value (equal to the number of master nodes, usually 1 or 3).

   Example for 1 replica:

   d8 k -n d8-stronghold scale statefulset stronghold --replicas=1
   

Disabling the module

You can disable the module using one of the following methods:

• Using the Deckhouse web interface.

  In the “System” → “System Management” → “Deckhouse” → “Modules” section, open the stronghold module and turn off the “Module Enabled” switch. Save the changes.

• Using Deckhouse CLI.

  Execute the following commands to disable the module:

  d8 k annotate mc stronghold modules.deckhouse.io/allow-disabling=true
  d8 system module disable stronghold
  

After disabling the module, if the old data is no longer needed, execute the following command to remove it on each master node of the cluster:

sudo rm -Ir /var/lib/deckhouse/stronghold/{node-id,vault.db,raft}

How to access the service

Access to the service is provided through inlets. Currently, only one inlet, Ingress, is available. The web interface address for Stronghold is formed as follows: in the template publicDomainTemplate of the Deckhouse global configuration parameter, the %s placeholder is replaced with stronghold keyword.

For example, if publicDomainTemplate is set to %s-kube.mycompany.tld, than the Stronghold web interface will be accessible at the address stronghold-kube.cmycompany.tld.

How to use Data Storage. Operating Modes

The data stored in Stronghold is encrypted. To decrypt the storage data, an encryption key is required. The encryption key is also stored with the data (as part of key bundles), but it is encrypted with another encryption key known as the root key.

To decrypt Stronghold data, it is necessary to decrypt the encryption key, which requires the root key. Unsealing the storage is the process of gaining access to this root key. The root key is stored along with all other storage data but is encrypted with another mechanism: the unseal key.

In the current version of the module, there is only the Automatic mode, in which the storage is automatically initialized during the first module launch. During the initialization process, the unlocking key and root token are both placed into the stronghold-keys secret in the d8-stronghold namespace in the Kubernetes cluster. After the initialization, the module automatically unseals the nodes of the Stronghold cluster. In the automatic mode, in the event of a restart of Stronghold nodes, the storage will also be automatically unsealed without manual intervention.

Access Management

The role named deckhouse_administrators is created after storage initialization using the Automatic mode of the Stronghold module. This role is granted access to the web interface through OIDC authentication via Dex. Additionally, the automatic connection of the current Deckhouse cluster to Stronghold is configured. This is necessary for the operation of the secrets-store-integration module.

To provide access to the users with the admins group membership (group membership is conveyed from the used IdP or LDAP via Dex), you need to specify this group in the administrators array in the ModuleConfig:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: stronghold
spec:
  enabled: true
  version: 1
  settings:
    management:
      mode: Automatic
      administrators:
      - type: Group
        name: admins

To grant administrator rights to users with roles manager and securityoperator, you can use the following parameters in the ModuleConfig:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: stronghold
spec:
  enabled: true
  version: 1
  settings:
    management:
      mode: Automatic
      administrators:
      - type: User
        name: manager@mycompany.tld
      - type: User
        name: securityoperator@mycompany.tld

If needed, you can create users in Stronghold with different access rights to secrets using the built-in storage mechanisms.

Running with a self-signed certificate

You need to create a CA, a certificate, and sign it with the created CA. If there is already a CA, you can sign the certificate using it. Note that you have to create a fullchain certificat.

The createCertificate.sh script below creates the proper certificate + key pair for the mycompany.tld domain (*.mycompany.tld) using openssl.

#!/bin/bash

set -e
caName="MyOrg-RootCA"            # CA name (CN)
publicDomain="mycompany.tld"     # Cluster domain name (see publicDomainTemplate)
certName="kubernetes"            # Cluster certificate name (CN)

mkdir -p "${caName}"
cd "${caName}"

[ ! -f "${caName}.key" ] && openssl genrsa -out "${caName}.key" 4096

[ ! -f "${caName}.crt" ] &&  openssl req -x509 -new -nodes -key "${caName}.key" -sha256 -days 1826 -out "${caName}.crt" \
   -subj "/CN=${caName}/O=MyOrganisation"

openssl req -new -nodes -out ${certName}.csr -newkey rsa:4096 -keyout "${certName}.key" \
  -subj "/CN=${certName}/O=MyOrganisation"

# v3 ext file
cat > "${certName}.v3.ext" << EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${publicDomain}
DNS.2 = *.${publicDomain}
EOF

openssl x509 -req -in "${certName}.csr" -CA "${caName}.crt" -CAkey "${caName}.key" -CAcreateserial -out "${certName}.crt" -days 730 -sha256 -extfile "${certName}.v3.ext"

cat "${certName}.crt" "${caName}.crt" > "${certName}_fullchain.crt"

You have to create a secret in the d8-system namespace using the generated kubernetes.key and kubernetes_fullchain.crt files:

d8 k -n d8-system create secret tls mycompany-wildcard-tls --cert=kubernetes_fullchain.crt --key=kubernetes.key

To use the created certificate in the cluster, you need to configure the global module as follows (use the d8 k edit mc global command):

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: global
spec:
  settings:
    modules:
      https:
        customCertificate:
          secretName: mycompany-wildcard-tls
        mode: CustomCertificate
      publicDomainTemplate: '%s.mycompany.tld'
  version: 2

Now you can enable the stronghold module. It will automatically initialize and set up integration with dex.

d8 system module enable stronghold

How to obtain the stronghold binary file

To obtain the stronghold binary file, execute the following command on the master node of the cluster as the root user:

mkdir $HOME/bin
sudo cp /proc/$(pidof stronghold)/root/usr/bin/stronghold $HOME/bin/stronghold && sudo chmod a+x $HOME/bin/stronghold
export PATH=$PATH:$HOME/bin

The executable file stronghold will be located in the /bin/stronghold folder of the user’s home directory.