Break-glass access

People usually log in to Stronghold through OIDC. If the identity provider (in DP, Dex and an external IdP) is unavailable or misconfigured, no administrator can log in. Break-glass access is a pre-arranged, rarely used, and tightly controlled way to log in in such a situation.

Goal

Prepare two levels of emergency access:

• Level 1: a separate userpass account with an administrative policy whose password is kept in a sealed envelope.
• Level 2: the generate-root procedure for when level 1 does not help.

Set up auditing of their use and a rotation procedure after use.

Prerequisites

Before you start, prepare the following:

• A Stronghold token with administrative permissions.
• Storage for envelopes (a safe) and at least two people responsible for access to it.
• Information about the holders of unseal or recovery key shares and their quorum threshold.
• For auditing: a configured audit device (see Audit).

Step 1. Create a break-glass policy

The policy must allow restoring login, but does not have to grant full access to secrets. An example policy for restoring auth methods and policies:

d8 stronghold policy write break-glass - <<'POLICY'
path "sys/auth" {
  capabilities = ["read"]
}
path "sys/auth/*" {
  capabilities = ["create", "read", "update", "delete", "sudo"]
}
path "auth/*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
path "sys/policies/acl/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "identity/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "sys/mounts" {
  capabilities = ["read"]
}
POLICY

Extend the policy only with what is actually needed in an emergency.

Step 2. Create a break-glass account

Create a separate userpass account for emergency login.

1. Enable a separate instance of the userpass method so that its use is easy to distinguish in audit logs:

   d8 stronghold auth enable -path=breakglass userpass

2. Generate a long random password on an isolated workstation:

   openssl rand -base64 32

3. Create a user with a short token lifetime:

   d8 stronghold write auth/breakglass/users/breakglass-1 password="<BREAKGLASS_PASSWORD>" token_policies=break-glass token_ttl=1h token_max_ttl=4h

   <BREAKGLASS_PASSWORD> is the password generated in the previous step.

4. Write the password on paper, put it in an envelope, seal and sign it, and place it in the safe. For a two-person rule, split the password into two parts and keep them in separate envelopes with different people.

5. Remove the password from the shell history and the clipboard.

Lockout after failed login attempts is enabled for userpass by default. An attacker can lock the break-glass account on purpose; take this into account in the procedure and, if needed, create a second breakglass-2 account in a separate envelope.

Step 3. Prepare the generate-root procedure

If the break-glass account is unavailable, a root token can be generated only with a quorum of unseal key share holders (with auto-unseal, recovery key share holders).

1. The initiator starts the procedure. The response contains nonce and otp:

   d8 stronghold operator generate-root -init

2. Each share holder submits their share, specifying the nonce:

   d8 stronghold operator generate-root -nonce=<NONCE>

   <NONCE> is the nonce value from the -init output.

   The command prompts for the key share interactively. When the quorum is reached, encoded_token is printed.

3. The initiator decodes the token:

   d8 stronghold operator generate-root -decode=<ENCODED_TOKEN> -otp=<OTP>

   Placeholders:

   • <ENCODED_TOKEN>: the encoded_token value printed when the quorum is reached;
   • <OTP>: the otp value from the -init output.

If the procedure was started by mistake, cancel it with d8 stronghold operator generate-root -cancel. Describe the procedure in your runbook, including who holds the shares and how to gather them.

In DP in Automatic mode, Stronghold is initialized with one unseal key share and a threshold of 1, so a single share is enough for the generate-root quorum. This share is stored in the unsealKey field of the stronghold-keys secret, and the initial root token is in the rootToken field. For generate-root, take the share from the secret (or from your copy of it) and enter it in step 2. After the procedure, store the new root token as an emergency one, and revoke it after use (d8 stronghold token revoke -self). The module uses the root token from stronghold-keys only to migrate the Kubernetes auth role when its own login to Stronghold fails; if the rootToken field is empty or the secret is missing, this step is skipped. Therefore, after the initial setup the root token can be revoked and removed from the secret.

Step 4. Set up usage monitoring

Every use of break-glass access must trigger an alert.

1. Make sure an audit device is enabled:

   d8 stronghold audit list

2. Configure an alert in your SIEM for any login through the break-glass path. Indicators in the audit log:

   • request.path equals auth/breakglass/login/breakglass-1;
   • request.path starts with sys/generate-root;
   • auth.policies contains root.

   An example search in a file audit log:

   jq -c 'select(.request.path | startswith("auth/breakglass/login") or startswith("sys/generate-root"))' /var/log/stronghold_audit.log

Use in an emergency

In an emergency, follow these steps:

1. Two responsible people open the envelope and record the time and reason in the incident log.

2. Log in:

   d8 stronghold login -method=userpass -path=breakglass username=breakglass-1

3. Restore the login configuration (for example, auth/oidc_deckhouse/config) and check that regular login works.

4. Revoke the break-glass token: d8 stronghold token revoke -self.

Rotation after use

After every use, including drills:

1. Change the password and create a new envelope:

   d8 stronghold write auth/breakglass/users/breakglass-1/password password="<NEW_PASSWORD>"

   <NEW_PASSWORD> is a new random password.

2. Revoke all tokens issued through the break-glass path:

   d8 stronghold lease revoke -prefix auth/breakglass/

3. If generate-root was used, revoke the root token as soon as the work is done:

   d8 stronghold token revoke <ROOT_TOKEN>

   <ROOT_TOKEN> is the root token obtained with generate-root.

4. Review the audit log for the period of use and attach an export to the incident report.

Verification

Run a drill at least every six months:

1. Log in with the break-glass account on a test or production cluster.
2. Check the permissions: d8 stronghold token capabilities sys/auth/oidc_deckhouse.
3. Make sure the SIEM received an alert.
4. Perform the rotation after use.

Cleanup

If break-glass access is no longer needed (for example, when decommissioning the cluster):

d8 stronghold auth disable breakglass
d8 stronghold policy delete break-glass

Destroy the password envelopes with a formal record.

Additional resources

• Userpass auth method
• Tokens
• Seal and unseal
• Audit
• Policies