How to run kube-bench in my cluster?

First, you have to exec in Deckhouse Pod:

kubectl -n d8-system exec -ti svc/deckhouse-leader -c deckhouse -- bash

Then you have to select which node you want to run kube-bench.

  • Run on random node:

    curl -s https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml | kubectl create -f -

  • Run on specific node, e.g. control-plane node:

    curl -s https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml | kubectl apply -f - --dry-run=client -o json | jq '.spec.template.spec.tolerations=[{"operator": "Exists"}] | .spec.template.spec.nodeSelector={"node-role.kubernetes.io/control-plane": ""}' | kubectl create -f -

Then you can check report:

kubectl logs job.batch/kube-bench

Deckhouse set the log retention period to 7 days. However, according to the security requirements specified in kube-bench, logs should be retained for at least 30 days. Use separate storage for logs if you need to keep logs for more than 7 days.

How to collect debug info?

We always appreciate helping users with debugging complex issues. Please follow these steps so that we can help you:

  1. Collect all the necessary information by running the following command:

    kubectl -n d8-system exec svc/deckhouse-leader -c deckhouse \
  -- deckhouse-controller collect-debug-info \
  > deckhouse-debug-$(date +"%Y_%m_%d").tar.gz

  2. Send the archive to the Deckhouse team for further debugging.

Data that will be collected:

Category Collected data
Deckhouse
  • Deckhouse queue state
  • Deckhouse values (except for kubeRBACProxyCA and registry.dockercfg)
  • Current version of the deckhouse Pod
  • All DeckhouseRelease objects
  • Logs of Deckhouse Pods
  • Manifests of controllers and Pods from all Deckhouse namespaces
Cluster objects All objects of the following resources:
  • NodeGroup
  • NodeGroupConfiguration
  • Node
  • Machine
  • Instance
  • StaticInstance
  • MachineDeployment
  • ClusterAuthorizationRule
  • AuthorizationRule
  • ModuleConfig
As well as Events from all namespaces
Modules and their states
  • List of enabled modules
  • List of ModuleSource objects in the cluster
  • List of ModulePullOverride objects in the cluster
  • List of modules in maintenance mode
Controller logs and manifests Logs of the following components:
  • machine-controller-manager
  • cloud-controller-manager
  • csi-controller
  • cluster-autoscaler
  • Vertical Pod Autoscaler admission controller
  • Vertical Pod Autoscaler recommender
  • Vertical Pod Autoscaler updater
YAML manifests of the following controllers:
  • capi-controller-manager
  • caps-controller-manager
  • machine-controller-manager
Monitoring and alerts
  • Prometheus logs
  • All active alerts in Prometheus
  • List of all Pods not in the Running state, except those in Completed or Evicted states
Network
  • All objects from the d8-istio namespace
  • All istio custom resources
  • Envoy configuration for istio
  • Logs of istio
  • Logs of the istio ingressgateway
  • Logs of the istio users
  • Cilium connection status (cilium health status)

How to debug pod problems with ephemeral containers?

Run the following command:

kubectl -n <namespace_name> debug -it <pod_name> --image=ubuntu <container_name>

More info in official documentation.

How to debug node problems with ephemeral containers?

Run the following command:

kubectl debug node/mynode -it --image=ubuntu

More info in official documentation.