The deckhouse module: FAQ

How to run kube-bench in my cluster?

First, you have to exec in Deckhouse Pod:

d8 k -n d8-system exec -ti svc/deckhouse-leader -c deckhouse -- bash

Then you have to select which node you want to run kube-bench.

• Run on random node:

  curl -s https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml | d8 k create -f -
  

• Run on specific node, e.g. control-plane node:

  curl -s https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml | d8 k apply -f - --dry-run=client -o json | jq '.spec.template.spec.tolerations=[{"operator": "Exists"}] | .spec.template.spec.nodeSelector={"node-role.kubernetes.io/control-plane": ""}' | d8 k create -f -
  

Then you can check report:

d8 k logs job.batch/kube-bench

How to collect debug info?

1. Create a diagnostic archive with the d8 utility, redirecting its output (stdout) to a file:

   d8 system collect-debug-info > deckhouse-debug-$(date +"%Y_%m_%d").tar.gz
   

2. Send the resulting archive to the Deckhouse team for further debugging.

The --exclude flag omits the specified items from the archive. Example:

  d8 system collect-debug-info --exclude=queue global-values > deckhouse-debug-$(date +"%Y_%m_%d").tar.gz

The --list-exclude flag prints the list of items available for exclusion. Example:

  d8 system collect-debug-info --list-exclude

The following information is produced when creating the archive. Names in the “File in archive” column correspond to top-level items inside the resulting tar.gz archive. Certain sensitive values (e.g., kubeRBACProxyCA and registry.dockercfg) are excluded.

How to debug pod problems with ephemeral containers?

Run the following command:

d8 k -n <namespace_name> debug -it <pod_name> --image=ubuntu <container_name>

More info in official documentation.

An ephemeral container cannot be added to a Pod labeled heritage: deckhouse, see the next section.

Who can exec into Deckhouse Pods and scale Deckhouse objects?

Only Deckhouse system users. The deckhouse module denies the following operations to everyone else, cluster administrators and superadmins included, in every namespace:

• kubectl exec, kubectl attach, kubectl cp, kubectl port-forward and kubectl debug into a Pod labeled heritage: deckhouse. A port-forward to a Service ends up in one of its Pods and is denied the same way;
• kubectl scale of a Deployment, StatefulSet, ReplicaSet or ReplicationController labeled heritage: deckhouse, unless the object is under maintenance, that is, labeled maintenance.deckhouse.io/no-resource-reconciliation.

The label is checked on the object itself: a Pod without heritage: deckhouse is not protected, even if its Deployment carries the label.

The checks do not apply to:

• system:sudouser;
• dhctl and observability;
• service accounts from d8-* and kube-* namespaces (system:serviceaccount:d8-*, system:serviceaccount:kube-*), the Deckhouse controllers and the horizontal pod autoscaler of kube-controller-manager among them;
• the Kubernetes control plane components (system:apiserver, system:kube-controller-manager, system:kube-scheduler, system:volume-scheduler) and nodes (the system:nodes group);
• the system:masters group, that is, super-admin.conf on a master node. admin.conf (the kubeadm:cluster-admins group) is subject to the checks.

Apart from system:masters, these are the users who may also change objects labeled heritage: deckhouse. The checks do not depend on the admission-policy-engine module.

The checks run in the webhook-handler Pods of the d8-system namespace and fail closed. While webhook-handler is unavailable, kubectl exec, attach, cp, port-forward into any Pod and kubectl scale of any Deployment, StatefulSet, ReplicaSet or ReplicationController are denied in every namespace for everyone but the users listed above, whether or not the target is labeled. To get in for diagnostics, run the command as system:sudouser in the system:sudousers group, which needs the right to impersonate both, or use super-admin.conf on a master node. The group is needed for exec because the user-authz module checks exec on the same webhook-handler and exempts the group rather than the user:

d8 k --as=system:sudouser --as-group=system:sudousers -n d8-system exec -it <pod_name> -- sh

How to debug node problems with ephemeral containers?

Run the following command:

d8 k debug node/mynode -it --image=ubuntu

More info in official documentation.