The operator-trivy module: CIS Kubernetes Benchmark checks

The module performs automated cluster security checks in line with the CIS Kubernetes Benchmark.

Version in use

The module implements checks according to CIS Kubernetes Benchmark v1.12 (canonical mapping for Kubernetes 1.32–1.34, also applied to 1.35). Earlier DKP releases shipped v1.23, which actually maps to CIS Kubernetes Benchmark v1.5.0 (Kubernetes 1.22 era).

Component versions:

d8 k -n d8-operator-trivy get pods -o json | \
  jq -r '.items[] | .metadata.name as $p | .spec.containers[] | "\($p)\t\(.name)\t\(.image)"'

Check categories

CIS Kubernetes Benchmark checks are grouped into the following categories:

Exclusions

In Deckhouse Kubernetes Platform (DKP), some checks are disabled or excluded from reports. This is due to platform architecture and system component requirements.

Globally disabled checks

The following checks are disabled for all cluster resources:

CIS 1.2.1 — Anonymous auth (AVD-KCV-0001)

Check: Ensure that the --anonymous-auth argument is set to false.

Status: Completely disabled.

Reason: The Trivy/defsec CIS 1.2.1 implementation is based on inspecting the legacy --anonymous-auth flag. On newer Kubernetes versions, anonymous authentication configuration may be managed differently (for example, via AuthenticationConfiguration), therefore a flag-based check may produce false FAIL/WARN results (including in kube-bench). For auditing, rely on the actual kube-apiserver configuration and access controls in your environment.

CIS 1.2.13 — SecurityContextDeny (AVD-KCV-0013)

Check: Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used.

Status: Completely disabled.

Reason: The SecurityContextDeny admission controller was deprecated and completely removed in Kubernetes v1.30. This controller blocked pods with privileged settings, but its approach was too coarse and didn’t allow granular policy configuration.

Alternative in DKP: Deckhouse uses Pod Security Standards implemented in the admission-policy-engine module based on OPA Gatekeeper. This allows:

• Applying Privileged, Baseline, or Restricted policies at namespace level
• Creating exceptions for specific workloads
• Flexible rule configuration per organization requirements

Checks disabled for system namespaces

The following checks are disabled for kube-system and d8-* namespaces (all Deckhouse namespaces start with d8- prefix).

Why do system components need privileges?

A Kubernetes cluster requires components that work at a low level: network management, storage, host resource monitoring. These components by their nature cannot run in an isolated container without privileges.

Partial exclusions

Specific exclusions apply to individual resources:

CIS 5.7.4 — Default namespace (AVD-KSV-0110)

Check: The default namespace should not be used.

Exclusion: Resource default/service-kubernetes.

Reason: The kubernetes service in the default namespace is a built-in Kubernetes API Server service. It’s created automatically during cluster initialization and cannot be moved to another namespace. This is standard Kubernetes behavior documented in official documentation.

CIS 5.7.3 — Security Context (AVD-KSV-0020, AVD-KSV-0021)

Check: Apply Security Context to Your Pods and Containers.

Exclusion: ReplicaSets in d8-* namespaces.

Reason: Checks require runAsUser > 10000 and runAsGroup > 10000 (using unprivileged UID/GID). In DKP there are system workloads that run as root (runAsUser/runAsGroup = 0) or do not set runAsUser/runAsGroup explicitly — this can be required for host filesystem compatibility and low-level operations.

• Compatibility with host file permissions
• Correct operation with volumes where permissions are already configured
• Alignment with standard system service practices

How to check which objects are “below the 5.7.3 threshold” in your cluster (readable format):

Short summary (pod-level runAsUser/runAsGroup) for system namespaces d8-*, kube-system, kube-public, kube-node-lease:

d8 k get deploy,rs,ds,sts -A -o json | jq -r '
  .items[]
  | select(.metadata.namespace | test("^(d8-)|^(kube-system|kube-public|kube-node-lease)$"))
  | . as $o
  | ($o.spec.template.spec.securityContext.runAsUser // null) as $u
  | ($o.spec.template.spec.securityContext.runAsGroup // null) as $g
  | select((($u|tonumber? // -1) <= 10000) or (($g|tonumber? // -1) <= 10000))
  | [
      $o.kind,
      $o.metadata.namespace,
      $o.metadata.name,
      ("runAsUser=" + (($u|tostring) // "null")),
      ("runAsGroup=" + (($g|tostring) // "null"))
    ] | @tsv'

Container-level detail to see what is set on pod vs container level:

d8 k get deploy,rs,ds,sts -A -o json | jq -r '
  .items[]
  | select(.metadata.namespace | test("^(d8-)|^(kube-system|kube-public|kube-node-lease)$"))
  | . as $o
  | ($o.spec.template.spec.securityContext.runAsUser // null) as $pu
  | ($o.spec.template.spec.securityContext.runAsGroup // null) as $pg
  | $o.spec.template.spec.containers[]?
  | . as $c
  | ($c.securityContext.runAsUser // null) as $cu
  | ($c.securityContext.runAsGroup // null) as $cg
  | select((($pu|tonumber? // 999999) <= 10000) or (($pg|tonumber? // 999999) <= 10000) or (($cu|tonumber? // 999999) <= 10000) or (($cg|tonumber? // 999999) <= 10000))
  | [
      $o.kind,
      $o.metadata.namespace,
      $o.metadata.name,
      ("pod.runAsUser=" + (($pu|tostring) // "null")),
      ("pod.runAsGroup=" + (($pg|tostring) // "null")),
      ("container=" + $c.name),
      ("container.runAsUser=" + (($cu|tostring) // "null")),
      ("container.runAsGroup=" + (($cg|tostring) // "null"))
    ] | @tsv'

Checks excluded from metrics

The following checks are performed, but their results are not displayed in Prometheus metrics or on the Grafana dashboard. This is intentional so the dashboard reflects real security issues, not false positives.

CIS 5.1.2 — Access to secrets (AVD-KSV-0041)

Check: Minimize access to secrets.

What it checks: Finds all Roles/ClusterRoles with get, list, or watch permissions on the secrets resource.

Why excluded from metrics:

In the original CIS Benchmark, this check is marked as type: manual and scored: false because:

• It’s impossible to automatically determine if secrets access is legitimate
• Any controller working with TLS or credentials requires such access

Examples of roles with legitimate secrets access in DKP:

• d8:ingress-nginx:kruise-role — managing secrets for ingress controllers
• d8:node-manager:machine-controller-manager — role for machine-controller-manager (namespace d8-cloud-instance-manager) reads secrets/configmaps
• d8:operator-prometheus — Prometheus operator ClusterRole reads/creates secrets
• Role/d8-log-shipper/log-shipper — log-shipper Role reads secrets in d8-log-shipper

How to see the full list of roles with secrets access:

# All Roles/ClusterRoles that have secrets access (get/list/watch)
d8 k get clusterroles,roles -A -o json | jq -r '
  .items[]
  | select(any(.rules[]?;
      ((.resources // []) | index("secrets")) != null
      and (
        ((.verbs // []) | index("get")) != null
        or ((.verbs // []) | index("list")) != null
        or ((.verbs // []) | index("watch")) != null
      )
    ))
  | "\(.kind)\t\(.metadata.namespace // "-")\t\(.metadata.name)"'

# Quick lookup of bindings by role name (example)
role_name="d8:operator-prometheus"
d8 k get clusterrolebindings,rolebindings -A | grep -F "$role_name"

How to analyze results:

# Get all roles with secrets access
d8 k get clustercompliancereports.aquasecurity.github.io cis -ojson | \
  jq '.status.detailReport.results | map(select(.id == "5.1.2")) | .[].checks'

Analyze the list and ensure that:

• All roles belong to known system components
• No user roles have excessive access

CIS 5.1.3 — Wildcard use in Roles (AVD-KSV-0044, AVD-KSV-0045, AVD-KSV-0046)

Check: Minimize wildcard use in Roles and ClusterRoles.

What it checks: Finds roles with wildcards (*) in resources, verbs, or apiGroups fields.

Why excluded from metrics:

In the original CIS Benchmark, this check is marked as type: manual and scored: false because:

• cluster-admin — standard Kubernetes role, contains wildcards
• Operators often require broad access to manage diverse resources

Examples of legitimate wildcard use in DKP:

• cluster-admin — full access for administrators
• deckhouse ClusterRole — managing all Deckhouse modules
• CRD operators — access to all resources in their API group

How to analyze results:

# Get all roles with wildcards
d8 k get clustercompliancereports.aquasecurity.github.io cis -ojson | \
  jq '.status.detailReport.results | map(select(.id == "5.1.3")) | .[].checks'

Ensure wildcards are used only in:

• Kubernetes system roles (cluster-admin, admin, edit, view)
• Deckhouse component roles (d8-*)
• Known operator roles

Known specifics

Kubelet TLS certificate check (CIS 4.2.10)

Checks AVD-KCV-0088 and AVD-KCV-0089 may show FAIL status. This is expected behavior: Deckhouse uses automatic kubelet certificate rotation (RotateKubeletServerCertificate) instead of static --tls-cert-file and --tls-private-key-file files.

You can read more in the control-plane-manager module FAQ section.

CNI file checks (CIS 1.1.9 / 1.1.10)

Checks AVD-KCV-0056 and AVD-KCV-0057 (CNI configuration file permissions / ownership) target the CNI config directory /etc/cni/net.d. The upstream trivy-checks bundle ships an audit command with the glob /*/cni/*, which also covers the CNI plugin binaries in /opt/cni/bin (must be 755 to be executable) and the IPAM state under /var/lib/cni/networks (regular files without secrets). On any standard cluster that produces a FAIL with no actionable file path.

DKP ships a local override (images/operator/overrides/commands/kubernetes/containerNetworkInterfaceFile*.yaml) that scopes the audit to /etc/cni/net.d only, so the check reflects the real CIS recommendation.

AlwaysPullImages admission plugin (CIS 1.2.12)

Check AVD-KCV-0012 historically requires --enable-admission-plugins=AlwaysPullImages on kube-apiserver. That admission plugin defended against the “image cache leak” scenario: a pod in namespace B reusing an image that was originally pulled by a pod in namespace A with private registry credentials, since the kubelet’s local image cache makes the re-pull unnecessary.

Starting with Kubernetes 1.35 the same threat is mitigated at the kubelet layer by the KubeletEnsureSecretPulledImages feature gate (alpha in 1.33–1.34, beta-default in 1.35+). The kubelet re-verifies image-pull credentials against the requesting pod’s secrets even for pre-pulled images, without the constant re-pull traffic that AlwaysPullImages produces. Deckhouse supports Kubernetes 1.31–1.35; the gate is on by default on 1.35.

DKP ships a local override (images/operator/overrides/policies/kubernetes/policies/apiserver_always_pull_images_plugin.rego) that PASSes the check on Kubernetes 1.35+ regardless of the apiserver admission plugin, and falls back to the original requirement on 1.31–1.34. The Kubernetes minor version is parsed from the kube-apiserver container image tag (e.g. :v1.35.3). If the tag isn’t recognized the override is transparent and the original enforcement applies.

The control’s display name and description in the CIS report are updated correspondingly to “Ensure that image pull credentials are verified per pod (AlwaysPullImages or KubeletEnsureSecretPulledImages)”.

Manual checks

The following checks require manual verification and are not automated. This aligns with the original CIS Benchmark specification where they are marked as Manual.

Deckhouse customizations

DKP’s CIS report does not match what trivy-operator alone produces. The upstream trivy-checks bundle is frozen on k8s-cis-1.23; without local intervention you’d be scanning your cluster against a four-year-old spec. The module ships a small collection of overrides that close the gap. Each override is plain YAML / Rego in images/operator/overrides/ and is overlaid on top of the upstream bundle at build time — readable, reviewable, and easy to drop the day upstream ships the same fix.

What’s customised

• Spec bumped to v1.12. templates/reports/cis.yaml is the source of truth for our ClusterComplianceReport. It enumerates 131 controls in the v1.12 ordering (vs 116 in v1.23), preserves the AVD-KCV / CMD identifiers from upstream where the underlying audit logic didn’t change, and renumbers controls across renames (e.g. CIS section 5.7 became 5.6 in v1.12).
• CIS 1.1.9 / 1.1.10 — CNI file checks. Upstream’s audit glob /*/cni/* matches plugin binaries in /opt/cni/bin and IPAM state in /var/lib/cni/networks alongside the actual CNI config in /etc/cni/net.d. The override scopes the audit to /etc/cni/net.d only. Details in CNI file checks.
• CIS 1.2.11 — AlwaysPullImages. The original requirement (the AlwaysPullImages admission plugin) is satisfied at the kubelet layer by the KubeletEnsureSecretPulledImages feature gate (beta-default in Kubernetes 1.35). The override accepts either path. Details in AlwaysPullImages admission plugin.
• CIS 4.3.1 — kube-proxy metrics localhost. A new automated control in v1.12 that upstream trivy-checks doesn’t implement. DKP ships AVD-KCV-0801 in its private 0801..0899 range; the Rego fires only when kube-proxy is present and --metrics-bind-address resolves to a non-localhost address. With Cilium kubeProxyReplacement (the Deckhouse default on Cilium) there is no kube-proxy DaemonSet at all, so the rule never fires and the check passes naturally.
• tools/cis-converter. A one-shot Python script that bumps templates/reports/cis.yaml against the matching aquasecurity/kube-bench profile (cfg/cis-X.Y). It preserves AVD-KCV identifiers and Russian translations across renames; flagged collisions and a per-version rename map under tools/cis-converter/renames/ keep cascade-renumbering safe. Re-run it the day a new CIS version drops in kube-bench.

Reserved identifier ranges

To avoid colliding with future upstream additions, DKP-owned checks live in dedicated ranges:

Verifying with kube-bench yourself

If you want a second opinion against the canonical CIS audit, you can run kube-bench as a one-shot Kubernetes Job alongside the cluster. kube-bench is from Aqua Security too and tracks new CIS versions sooner than trivy-checks does, so it’s the closest thing to a ground-truth comparison.

A minimal master-node run:

apiVersion: batch/v1
kind: Job
metadata:
  name: kube-bench-master
  namespace: kube-system
spec:
  template:
    spec:
      hostPID: true
      nodeSelector:
        node-role.kubernetes.io/control-plane: ""
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        operator: Exists
      containers:
      - name: kube-bench
        image: aquasec/kube-bench:latest
        # Override the profile if your kube-bench autodetection picks
        # something other than what the report header says.
        command: ["kube-bench", "--benchmark", "cis-1.12", "run", "--targets", "master,etcd,controlplane,policies"]
        volumeMounts:
        - { name: var-lib-etcd,    mountPath: /var/lib/etcd,    readOnly: true }
        - { name: var-lib-kubelet, mountPath: /var/lib/kubelet, readOnly: true }
        - { name: etc-systemd,     mountPath: /etc/systemd,     readOnly: true }
        - { name: etc-kubernetes,  mountPath: /etc/kubernetes,  readOnly: true }
        - { name: usr-bin,         mountPath: /usr/local/mount-from-host/bin, readOnly: true }
      restartPolicy: Never
      volumes:
      - { name: var-lib-etcd,    hostPath: { path: /var/lib/etcd } }
      - { name: var-lib-kubelet, hostPath: { path: /var/lib/kubelet } }
      - { name: etc-systemd,     hostPath: { path: /etc/systemd } }
      - { name: etc-kubernetes,  hostPath: { path: /etc/kubernetes } }
      - { name: usr-bin,         hostPath: { path: /usr/bin } }
  backoffLimit: 0

d8 k apply -f kube-bench-master.yaml
d8 k -n kube-system logs job/kube-bench-master | tee kube-bench-master.txt

For workers, swap the nodeSelector and target list (--targets node) and apply it as a DaemonSet (or a Job per node). Aqua publishes ready-made manifests for both modes in kube-bench/job.yaml and kube-bench/hostPID/job.yaml.

Comparing kube-bench and DKP output

kube-bench and the DKP CIS report use the same control IDs (1.1.9, 4.3.1, …) so a side-by-side comparison is straightforward. Expect a few intentional divergences:

If you observe a divergence not in this table, it’s a candidate bug — open an issue against operator-trivy with both reports attached.

Mapping to kube-bench

DKP uses Trivy Operator for CIS checks instead of the kube-bench utility. Check mappings:

Use CIS Control ID (e.g., 1.2.1, 5.2.2) to compare results.

Viewing results

Grafana

Results are available on the Security / CIS Kubernetes Benchmark dashboard.

Command line

Get all results:

d8 k get clustercompliancereports.aquasecurity.github.io cis -ojson | \
  jq '.status.detailReport.results'

Get only failed checks:

d8 k get clustercompliancereports.aquasecurity.github.io cis -ojson | \
  jq '.status.detailReport.results | map(select(.checks | map(.success) | all | not))'

Get results for a specific check:

d8 k get clustercompliancereports.aquasecurity.github.io cis -ojson | \
  jq '.status.detailReport.results | map(select(.id == "5.7.3"))'

Check schedule

CIS Benchmark checks are performed:

• every 6 hours (cron: 0 */6 * * *);
• on operator startup.