TLS certificates

This page describes replacing Stronghold’s own TLS certificates: the API listener certificate and the certificates nodes use to communicate within the Raft cluster. Issuing certificates for applications is covered in PKI.

Monitor certificate expiry with the alerts from Monitoring and replace certificates in advance.

Stronghold in Linux

The listener certificate and key are set with the tls_cert_file and tls_key_file parameters of the listener "tcp" block. The certificate file must contain the full chain: the server certificate first, followed by the intermediate CA certificates.

Checking the current certificate

To view the subject, issuer, validity period and SANs of the current certificate, run the following command:

openssl x509 -in /opt/stronghold/tls/node-1-cert.pem -noout -subject -issuer -dates -ext subjectAltName

Replacing a certificate without a restart

Stronghold rereads the listener certificate and key files on SIGHUP. Changes to file paths are not applied this way, so place the new files at the same paths.

Perform on each node in turn:

1. Issue a new certificate with the same SANs (the node DNS names and IP addresses, and the load balancer address, if used).

2. Check that the certificate matches the key:

   openssl x509 -in node-1-cert.pem -noout -pubkey | sha256sum
   openssl pkey -in node-1-key.pem -pubout | sha256sum

   The hashes must match.

3. Back up the current files and replace them, preserving the owner and permissions:

   cp -a /opt/stronghold/tls /opt/stronghold/tls.bak-$(date +%F)
   install -o stronghold -g stronghold -m 0600 node-1-key.pem /opt/stronghold/tls/node-1-key.pem
   install -o stronghold -g stronghold -m 0644 node-1-cert.pem /opt/stronghold/tls/node-1-cert.pem

4. Send SIGHUP to the process:

   systemctl reload stronghold

5. Check that the server presents the new certificate:

   openssl s_client -connect raft-node-1.demo.tld:8200 </dev/null 2>/dev/null | openssl x509 -noout -dates

6. Check the logs for certificate reload errors:

   journalctl -u stronghold.service --since "5 minutes ago"

If the key is password-protected, the password must match the original one when reloading on SIGHUP.

The retry_join certificates (leader_client_cert_file, leader_ca_cert_file, leader_client_key_file) are not reread on SIGHUP: they are loaded when the node attempts to join the cluster, so new files take effect after the node restarts. Only the TLS listener certificates are reloaded on SIGHUP.

Replacing the CA

When changing the CA that issued node certificates, follow this order so that nodes and clients keep trusting each other:

1. Add the new CA to the trust store on all clients and to the leader_ca_cert_file files on all nodes (the file may contain several CA certificates).
2. Restart the nodes one at a time, starting with standby nodes, and wait until each node rejoins the cluster (d8 stronghold operator raft list-peers). Nodes with Shamir seal must be unsealed after a restart.
3. Replace node certificates with ones issued by the new CA as described above.
4. After all nodes are updated, remove the old CA from the trust stores.

A restart is required: the leader_ca_cert_file files are read only when the node joins.

Stronghold in DP

In DP, the following external access types (inlets) are supported: Ingress, GatewayAPI, LoadBalancer, NodePort and None. For LoadBalancer, NodePort and None, the https.mode: CustomCertificate mode is required. The certificate for the stronghold.<domain> domain is issued and stored according to the https settings described in Configuring Stronghold.

Stronghold uses two kinds of certificates:

• External certificate: the ingress-tls secret in CertManager mode or the ingress-tls-customcertificate secret in CustomCertificate mode, in the d8-stronghold namespace. It is mounted into the pod at /stronghold/tls and used by the API listener on port 8200.
• Internal certificate: the stronghold-tls secret (ca.crt, tls.crt, tls.key). It is issued by a module hook as a self-signed one and mounted at /stronghold/tls-internal. It is used on ports 8300, 8301 and 8500 (communication between pods and with the proxy). Its SANs include 127.0.0.1, stronghold, *.stronghold-internal, stronghold.d8-stronghold and stronghold.d8-stronghold.svc. It is managed by the module; you do not need to replace it manually.

In CertManager mode, the module creates a Certificate in the d8-stronghold namespace: stronghold with the Ingress inlet or stronghold-httproute with the GatewayAPI inlet. Both write the certificate to the ingress-tls secret. With other inlets, there is no Certificate.

Checking the certificate

To check the Certificate state and the validity period of the certificate, run the following commands:

d8 k -n d8-stronghold get certificate
d8 k -n d8-stronghold get secret ingress-tls -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject -issuer -dates

The commands are for the CertManager mode. In CustomCertificate mode, use the ingress-tls-customcertificate secret (there is no Certificate).

cert-manager certificates

cert-manager renews the certificate automatically before it expires. If the certificate was not renewed, check the CertificateRequest objects:

d8 k -n d8-stronghold get certificaterequest
d8 k -n d8-stronghold describe certificaterequest <NAME>

To force reissuing, delete the ingress-tls secret: cert-manager issues it again.

Custom certificate

In CustomCertificate mode, replace the contents of the secret in the d8-system namespace whose name is set in the https.customCertificate.secretName parameter of ModuleConfig stronghold. If the module https parameter is not set, the global modules.https settings of Deckhouse apply, and the secret name is taken from them:

d8 k -n d8-system create secret tls <SECRET_NAME> --cert=kubernetes_fullchain.crt --key=kubernetes.key --dry-run=client -o yaml | d8 k apply -f -

In this command, <SECRET_NAME> is the secret name, kubernetes_fullchain.crt is the file with the certificate chain, and kubernetes.key is the key file.

Deckhouse copies the updated certificate to module namespaces. Check that Stronghold presents the new certificate:

openssl s_client -connect stronghold.mycompany.tld:443 </dev/null 2>/dev/null | openssl x509 -noout -dates

If the CA has changed and the user-authn module uses dexCAMode: FromIngressSecret, make sure Dex has received the new CA, otherwise OIDC login stops working.

The module does not restart the stronghold pods when the certificate secret changes (the StatefulSet has checksum annotations only for the configuration and the license), and Stronghold rereads listener certificates only on SIGHUP. If the pod itself terminates TLS on port 8200 (the LoadBalancer and NodePort inlets), restart the pods one at a time or send SIGHUP to the process after the secret is updated.