The operator-trivy module: CIS Kubernetes Benchmark checks

The module performs automated cluster security checks in line with the CIS Kubernetes Benchmark.

Version in use

The module implements checks according to CIS Kubernetes Benchmark v1.12 (canonical mapping for Kubernetes 1.32–1.34, also applied to 1.35).

The upstream aquasecurity/trivy-checks bundle contains the k8s-cis-1.23 specification. DP adds the controls of CIS Kubernetes Benchmark v1.12 and changes some checks, as described in DP customizations.

The module uses the following component versions:

The table lists the versions of the current module release. To see the images that are running in a particular cluster, run:

d8 k -n d8-operator-trivy get pods -o json | jq -r '.items[] | .metadata.name as $p | .spec.containers[] | "\($p)\t\(.name)\t\(.image)"'

Check categories

CIS Kubernetes Benchmark checks are grouped into the following categories:

Exclusions

In DP, some checks are disabled or excluded from reports. This is due to platform architecture and system component requirements.

Namespaces and objects excluded from reports

The complianceReports.skipSystemResources parameter excludes platform objects from compliance reports. With the default value true, the module applies the following filters:

• ConfigAuditReport and RbacAssessmentReport are not created in the d8-*, kube-* and default namespaces.
• Cluster-scoped objects with the heritage=deckhouse or kubernetes.io/bootstrapping=rbac-defaults label are skipped, for example DP ClusterRoles and the built-in Kubernetes roles such as cluster-admin.

InfraAssessmentReport is not affected, so the checks of CIS section 1.x keep covering the control plane components in kube-system. Because nothing in the default namespace is evaluated, control 5.6.4 (default namespace) finds no objects there.

To exclude a namespace from compliance reports regardless of the parameter value, add the security.deckhouse.io/skip-compliance=true label to it:

d8 k label namespace <NAMESPACE> security.deckhouse.io/skip-compliance=true

<NAMESPACE> is the name of the namespace. ConfigAuditReport and RbacAssessmentReport are not created in a namespace with this label.

The exclusions for system namespaces and for the default namespace below, as well as the examples of DP roles, matter when complianceReports.skipSystemResources is false.

Globally disabled checks

The following check is disabled for all cluster objects.

CIS 1.2.1 — Anonymous auth (AVD-KCV-0001)

Check: Ensure that the --anonymous-auth argument is set to false.

Status: Completely disabled.

Reason: The Trivy/defsec CIS 1.2.1 implementation is based on inspecting the legacy --anonymous-auth flag. On newer Kubernetes versions, anonymous authentication configuration may be managed differently (for example, via AuthenticationConfiguration), therefore a flag-based check may produce false FAIL/WARN results (including in kube-bench). For auditing, rely on the actual kube-apiserver configuration and access controls in your environment.

Checks disabled for system namespaces

The following checks are disabled for objects in the kube-system namespace and in namespaces with the d8- prefix, which DP modules use. In other namespaces, these checks run in full and help identify insecure configurations.

A Kubernetes cluster requires components that work at a low level: network management, storage, host resource monitoring. These components cannot run in an isolated container without privileges. The table lists the checks and examples of DP objects that need the corresponding privileges:

Partial exclusions

The following exclusions apply to specific objects.

CIS 5.6.4 — Default namespace (AVD-KSV-0110)

Check: The default namespace should not be used.

Exclusion: Service kubernetes in the default namespace (default/service-kubernetes).

Reason: The kubernetes Service in the default namespace is the built-in Kubernetes API server Service. Kubernetes creates it during cluster initialization, and it cannot be moved to another namespace.

Security context (AVD-KSV-0020, AVD-KSV-0021)

Checks: AVD-KSV-0020 (UID 10000 or less) and AVD-KSV-0021 (GID 10000 or less) in ConfigAuditReports. They are not part of the CIS report: CIS 5.6.3, which covers the security context, is a manual control, see Manual checks.

Exclusion: AVD-KSV-0020 and AVD-KSV-0021 for ReplicaSets in d8-* namespaces.

Reason: These checks require runAsUser and runAsGroup greater than 10000. Some DP system workloads run as root or do not set runAsUser and runAsGroup, because they work with host files and with volumes whose permissions are already set.

The exclusion matters only when complianceReports.skipSystemResources is false. With the default value, reports are not created in d8-* namespaces at all, see Namespaces and objects excluded from reports.

To list the workloads in system namespaces whose pod or container sets runAsUser or runAsGroup to 10000 or less, run:

d8 k get deploy,rs,ds,sts -A -o json | jq -r '
  .items[]
  | select(.metadata.namespace | test("^(d8-)|^(kube-system|kube-public|kube-node-lease)$"))
  | . as $o
  | ($o.spec.template.spec.securityContext.runAsUser // null) as $pu
  | ($o.spec.template.spec.securityContext.runAsGroup // null) as $pg
  | $o.spec.template.spec.containers[]?
  | . as $c
  | ($c.securityContext.runAsUser // null) as $cu
  | ($c.securityContext.runAsGroup // null) as $cg
  | select((($pu|tonumber? // 999999) <= 10000) or (($pg|tonumber? // 999999) <= 10000) or (($cu|tonumber? // 999999) <= 10000) or (($cg|tonumber? // 999999) <= 10000))
  | [
      $o.kind,
      $o.metadata.namespace,
      $o.metadata.name,
      ("pod.runAsUser=" + (($pu|tostring) // "null")),
      ("pod.runAsGroup=" + (($pg|tostring) // "null")),
      ("container=" + $c.name),
      ("container.runAsUser=" + (($cu|tostring) // "null")),
      ("container.runAsGroup=" + (($cg|tostring) // "null"))
    ] | @tsv'

Checks excluded from metrics

The following checks are performed, but their results are not displayed in Prometheus metrics. This keeps the metrics focused on real security issues instead of false positives.

CIS 5.1.2 — Access to secrets (AVD-KSV-0041)

Check: Minimize access to secrets.

What it checks: Finds ClusterRoles that grant any of the get, list, watch, create, update, patch, delete, deletecollection, impersonate or * verbs on the secrets resource.

Why excluded from metrics: The CIS Benchmark marks this recommendation as Manual. Whether access to secrets is legitimate cannot be determined automatically: any controller that works with TLS certificates or credentials needs such access.

Examples of DP ClusterRoles with legitimate access to secrets:

• d8:ingress-nginx:kruise-role: Manages secrets for Ingress controllers.
• d8:node-manager:machine-controller-manager: Reads secrets and ConfigMaps for machine-controller-manager in the d8-cloud-instance-manager namespace.
• d8:operator-prometheus: Reads and creates secrets for the Prometheus operator.

To list the ClusterRoles with access to secrets, run:

d8 k get clusterroles -o json | jq -r '
  .items[]
  | select(any(.rules[]?;
      ((.resources // []) | index("secrets")) != null
      and any((.verbs // [])[];
        IN("get", "list", "watch", "create", "update", "patch", "delete", "deletecollection", "impersonate", "*"))
    ))
  | .metadata.name'

To list the bindings of a role, for example d8:operator-prometheus, run:

role_name="d8:operator-prometheus"
d8 k get clusterrolebindings,rolebindings -A | grep -F "$role_name"

To get the results of the control from the report, run:

d8 k get clustercompliancereports.trivy.deckhouse.io cis -ojson | jq '.status.detailReport.results | map(select(.id == "5.1.2")) | .[].checks'

Analyze the list and ensure that:

• All roles belong to known system components
• No user roles have excessive access

CIS 5.1.3 — Wildcard use in Roles (AVD-KSV-0044, AVD-KSV-0045, AVD-KSV-0046)

Check: Minimize wildcard use in Roles and ClusterRoles.

What it checks: Finds Roles and ClusterRoles that grant the * verb on all resources of all API groups or on sensitive resources such as secrets, pods and clusterroles, and ClusterRoles that grant access to all resources (*).

Why excluded from metrics: The CIS Benchmark marks this recommendation as Manual. The standard Kubernetes role cluster-admin contains wildcards, and operators often require broad access to manage diverse resources.

Examples of legitimate wildcard use in DP:

• cluster-admin: Full access for administrators.
• deckhouse ClusterRole: Management of all DP modules.
• CRD operators: Access to all resources in their API group.

To get the results of the control from the report, run:

d8 k get clustercompliancereports.trivy.deckhouse.io cis -ojson | jq '.status.detailReport.results | map(select(.id == "5.1.3")) | .[].checks'

Ensure wildcards are used only in the following roles:

• Kubernetes built-in roles such as cluster-admin
• Roles of DP components (ClusterRoles with the d8: prefix and Roles in d8-* namespaces)
• Known operator roles

Known specifics

The following checks behave differently from the upstream trivy-checks bundle or can fail on a DP cluster.

Kubelet TLS certificate check (CIS 4.2.9)

Checks AVD-KCV-0088 and AVD-KCV-0089 may show FAIL status. This is expected behavior: DP uses automatic kubelet certificate rotation (RotateKubeletServerCertificate) instead of static --tls-cert-file and --tls-private-key-file files. The details are in the control-plane-manager module FAQ.

CNI file checks (CIS 1.1.9 / 1.1.10)

Checks AVD-KCV-0056 and AVD-KCV-0057 (CNI configuration file permissions and ownership) target the CNI configuration directory /etc/cni/net.d. The upstream trivy-checks bundle audits the files that match /*/cni/*. This pattern also covers the CNI plugin binaries in /opt/cni/bin (they must be 755 to be executable) and the IPAM state in /var/lib/cni/networks (regular files without secrets). On a standard cluster, this produces a FAIL that does not point to any file.

DP overrides the upstream check so that the audit covers only /etc/cni/net.d, as the CIS recommendation requires.

AlwaysPullImages admission plugin (CIS 1.2.11)

The upstream check AVD-KCV-0012 requires --enable-admission-plugins=AlwaysPullImages on kube-apiserver. The admission plugin prevents a pod in namespace B from starting an image that a pod in namespace A pulled with private registry credentials: without it, the kubelet starts the image from its local cache and does not pull it again.

Kubernetes 1.35 mitigates the same threat in the kubelet with the KubeletEnsureSecretPulledImages feature gate (alpha in 1.33–1.34, beta and enabled by default in 1.35 and later). The kubelet verifies the image pull credentials of the requesting pod even for images that are already on the node, without the constant re-pull traffic that AlwaysPullImages causes. The Kubernetes versions that DP supports are listed in Supported versions.

DP overrides the check as follows:

• On Kubernetes 1.35 and later, the check passes without the admission plugin.
• On earlier versions, the check requires AlwaysPullImages in --enable-admission-plugins. Enabling the kubelet feature gate KubeletEnsureSecretPulledImages does not change the result.

The Kubernetes minor version is taken from the kube-apiserver container image tag (for example, :v1.35.3). If the tag does not contain a version, the check requires the admission plugin on any Kubernetes version.

In the CIS report, the control is named “Ensure that image pull credentials are verified per pod (AlwaysPullImages or KubeletEnsureSecretPulledImages)”.

etcd data directory ownership (CIS 1.1.12)

The upstream check AVD-KCV-0059 requires the etcd data directory /var/lib/etcd to be owned by etcd:etcd. In DP, the control plane runs as static pods under root, so /var/lib/etcd is owned by root:root. That is not a finding: root:root is at least as restrictive as etcd:etcd. Only root can read the database, and root is already the most privileged principal on the host.

Requiring etcd:etcd would add a dedicated etcd user and group on every master node with access to the directory that holds the whole secret store of the cluster (Secrets, tokens, the whole etcd keyspace), without improving confinement on a static-pod control plane.

DP overrides the check so that it passes when the data directory is owned by either root:root or etcd:etcd. In the CIS report, the control is named “Ensure that the etcd data directory ownership is set to root:root or etcd:etcd”.

Manual checks

The following controls are not automated and require manual verification. The CIS Benchmark marks them as Manual.

DP customizations

The CIS report in DP differs from the report of upstream Trivy Operator. The upstream trivy-checks bundle contains the k8s-cis-1.23 specification, and DP changes the specification and some of the checks on top of it.

What’s customized

DP makes the following changes:

• CIS Kubernetes Benchmark v1.12. The CIS report contains 131 controls numbered as in v1.12 (the upstream k8s-cis-1.23 specification has 116). Controls keep the upstream check identifiers where the check logic did not change. Control numbers differ from the upstream specification: for example, CIS section 5.7 became 5.6 in v1.12.
• CIS 1.1.9 / 1.1.10 — CNI file checks. The audit covers only /etc/cni/net.d. Details are in CNI file checks.
• CIS 1.2.11 — AlwaysPullImages. On Kubernetes 1.35 and later, the check passes without the admission plugin. On earlier versions, it requires AlwaysPullImages in --enable-admission-plugins; enabling the kubelet feature gate KubeletEnsureSecretPulledImages does not change the result. Details are in AlwaysPullImages admission plugin.
• CIS 1.1.12 — etcd data directory ownership. The check accepts root:root or etcd:etcd. Details are in etcd data directory ownership.
• CIS 4.3.1 — kube-proxy metrics bound to localhost. Upstream trivy-checks has no check for this v1.12 control. DP adds check AVD-KCV-0801, which fails when a kube-proxy container is started with --metrics-bind-address set to an address other than 127.0.0.1, localhost or [::1]. The check reads only the command-line flag. A metricsBindAddress set in the kube-proxy configuration file is not checked; verify it manually. With the cni-cilium module, Cilium replaces kube-proxy (kubeProxyReplacement), there is no kube-proxy DaemonSet, and the check passes.

Checks AVD-KCV-0801–AVD-KCV-0899 are DP checks that are not in upstream trivy-checks.

Verifying with kube-bench yourself

To compare the report with another implementation of the CIS audit, run kube-bench as a one-time Kubernetes Job. kube-bench is also developed by Aqua Security and has profiles for newer CIS versions than trivy-checks.

The following Job runs kube-bench on a master node:

apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench-master
  namespace: kube-system
spec:
  template:
    spec:
      hostPID: true
      nodeSelector:
        node-role.kubernetes.io/control-plane: ""
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
      containers:
      - name: kube-bench
        image: aquasec/kube-bench:latest
        command: ["kube-bench", "--benchmark", "cis-1.12", "run", "--targets", "master,etcd,controlplane,policies"]
        volumeMounts:
        - { name: var-lib-etcd,    mountPath: /var/lib/etcd,    readOnly: true }
        - { name: var-lib-kubelet, mountPath: /var/lib/kubelet, readOnly: true }
        - { name: etc-systemd,     mountPath: /etc/systemd,     readOnly: true }
        - { name: etc-kubernetes,  mountPath: /etc/kubernetes,  readOnly: true }
        - { name: usr-bin,         mountPath: /usr/local/mount-from-host/bin, readOnly: true }
      restartPolicy: Never
      volumes:
      - { name: var-lib-etcd,    hostPath: { path: /var/lib/etcd } }
      - { name: var-lib-kubelet, hostPath: { path: /var/lib/kubelet } }
      - { name: etc-systemd,     hostPath: { path: /etc/systemd } }
      - { name: etc-kubernetes,  hostPath: { path: /etc/kubernetes } }
      - { name: usr-bin,         hostPath: { path: /usr/bin } }
  backoffLimit: 0

Save the manifest as kube-bench-master.yaml and apply it:

d8 k apply -f kube-bench-master.yaml

Save the Job output to a file:

d8 k -n kube-system logs job/kube-bench-master | tee kube-bench-master.txt

To check worker nodes, change the nodeSelector and the target list (--targets node) and run the Job on each worker node. The kube-bench repository contains ready-made manifests job-master.yaml and job-node.yaml.

Comparing kube-bench and DP output

kube-bench and the DP CIS report use the same control IDs (1.1.9, 4.3.1 and so on), so you can compare them control by control. In DP, control plane and node checks (AVD-KCV-*) run commands on the nodes through k8s-node-collector, and policy checks (AVD-KSV-*) analyze Kubernetes objects.

The results differ for the following controls:

Viewing results

You can view the results in Deckhouse Console or with d8 k.

Deckhouse Console

CIS Benchmark results are available in System → System management → Security → Compliance Reports.

Command line

Get all results:

d8 k get clustercompliancereports.trivy.deckhouse.io cis -ojson | jq '.status.detailReport.results'

Get only failed checks:

d8 k get clustercompliancereports.trivy.deckhouse.io cis -ojson | jq '.status.detailReport.results | map(select(.checks | map(.success) | all | not))'

Get results for a specific check:

d8 k get clustercompliancereports.trivy.deckhouse.io cis -ojson | jq '.status.detailReport.results | map(select(.id == "5.6.4"))'

Check schedule

CIS Benchmark checks run on operator startup and on the schedule set by scanPeriods.complianceRescanCron (default 0 */6 * * *, every 6 hours).