The operator-trivy module: FAQ

Viewing resources that failed CIS compliance checks

The following command lists the CIS controls of ClusterComplianceReport cis that have at least one failed check:

d8 k get clustercompliancereports.trivy.deckhouse.io cis -ojson |
  jq '.status.detailReport.results | map(select(.checks | map(.success) | all | not))'

Viewing resources that have not passed a specific CIS compliance check

To view the results of one CIS control, filter ClusterComplianceReport cis by the control ID or by its description.

Filter by control ID:

check_id="5.6.4"
d8 k get clustercompliancereports.trivy.deckhouse.io cis -ojson |
  jq --arg check_id "$check_id" '.status.detailReport.results | map(select(.id == $check_id))'

Filter by control description:

check_desc="The default namespace should not be used"
d8 k get clustercompliancereports.trivy.deckhouse.io cis -ojson |
  jq --arg check_desc "$check_desc" '.status.detailReport.results | map(select(.description == $check_desc))'

Manual rescan of a resource

By default, the module rescans resources when a report becomes older than scanPeriods.workloadRescanPeriod (default 24h):

1. A VulnerabilityReport object is created in the namespace with each scanned resource.
2. This object contains the annotation trivy-operator.aquasecurity.github.io/report-ttl, which specifies the report lifetime (the default matches scanPeriods.workloadRescanPeriod).
3. After the lifetime expires, the object is deleted, which triggers a rescan of the resource.

Set scanPeriods.workloadRescanPeriod to "" to disable periodic rescanning (reports are generated once and are not refreshed automatically).

You can force a resource rescan in one of the following ways:

• Overwrite the annotation trivy-operator.aquasecurity.github.io/report-ttl, specifying a short report lifetime.
• Delete the VulnerabilityReport object from the namespace where the scanned resource is located.

Example command for overwriting the annotation trivy-operator.aquasecurity.github.io/report-ttl:

d8 k -n <NAMESPACE> annotate vulnerabilityreports.trivy.deckhouse.io <REPORT_NAME> trivy-operator.aquasecurity.github.io/report-ttl=1h --overwrite

Where:

• <NAMESPACE>: Namespace of the scanned resource.
• <REPORT_NAME>: Name of the VulnerabilityReport. To list the reports in the namespace, run d8 k -n <NAMESPACE> get vulnerabilityreports.trivy.deckhouse.io.

Manual rescan of a container registry repository

The module rescans images in a container registry repository according to the schedule defined in the rescanPeriod parameter of the RegistryScanTarget object.

To force an immediate rescan, remove the registry-scanner.deckhouse.io/last-scan-time annotation from the RegistryScanTarget object. The controller starts a new scan cycle immediately, without waiting for the next scheduled run.

Who has access to scan results

Access to scan results depends on the role-based model of the user-authz module used in the cluster.

In the granular role-based model, the following roles grant access to scan results:

• d8:use:role:viewer or higher, bound with a RoleBinding in a namespace: Reports in that namespace (VulnerabilityReport, SbomReport, ConfigAuditReport, ExposedSecretReport, InfraAssessmentReport, RbacAssessmentReport) and SecurityScore.
• d8:manage:security:viewer or higher, bound with a ClusterRoleBinding: Cluster-wide reports (ClusterComplianceReport and other Cluster* reports) and ClusterSecurityScore.
• d8:manage:permission:module:operator-trivy:view, bound with a ClusterRoleBinding: The same cluster-wide reports, without access to other modules of the security subsystem.

The d8:manage:* roles grant access to cluster-wide reports only. They do not grant access to NodeVulnerabilityReport and RegistryImageVulnerabilityReport: grant it with Kubernetes RBAC as shown in the example below.

In the basic role-based model (AuthorizationRule and ClusterAuthorizationRule), the following access levels grant access to scan results:

• User or higher, in an AuthorizationRule or ClusterAuthorizationRule: Namespaced reports.
• ClusterEditor or higher, in a ClusterAuthorizationRule: Cluster-wide reports and NodeVulnerabilityReport.

You can also grant access directly with standard Kubernetes RBAC (Role or ClusterRole with a RoleBinding or ClusterRoleBinding):

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: trivy-reports-view
  namespace: <NAMESPACE>
rules:
- apiGroups: ["trivy.deckhouse.io", "aquasecurity.github.io"]
  resources:
  - vulnerabilityreports
  - configauditreports
  - exposedsecretreports
  - sbomreports
  - infraassessmentreports
  - rbacassessmentreports
  verbs: ["get", "list", "watch"]
- apiGroups: ["deckhouse.io"]
  resources:
  - securityscores
  verbs: ["get", "list", "watch"]

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: trivy-cluster-reports-view
rules:
- apiGroups: ["trivy.deckhouse.io", "aquasecurity.github.io"]
  resources:
  - clustercompliancereports
  - clustervulnerabilityreports
  - clusterconfigauditreports
  - clusterinfraassessmentreports
  - clusterrbacassessmentreports
  - clustersbomreports
  - nodevulnerabilityreports
  verbs: ["get", "list", "watch"]
- apiGroups: ["deckhouse.io"]
  resources:
  - clustersecurityscores
  - registryimagevulnerabilityreports
  verbs: ["get", "list", "watch"]

Which workloads are scanned in a namespace

In a namespace labeled with security-scanning.deckhouse.io/enabled="", the operator scans the images of all workloads of the following kinds: Pod, ReplicaSet, ReplicationController, StatefulSet, DaemonSet, CronJob, and Job. The list of scanned workloads within a namespace cannot be limited.

This applies to image scanning. Configuration scanning does not depend on the label and covers every namespace, except for the ones excluded by complianceReports.skipSystemResources and the ones labeled with security.deckhouse.io/skip-compliance=true.

How to view the scan report for your application

To view the scan results of your application, open Deckhouse Console: System → System management → Security → Vulnerabilities → By workloads. You can filter the results by the desired namespace and resource.

The module also stores the results in report objects created for each scanned object. The reference describes the naming structure and location of the reports.

How do the module’s CIS checks relate to kube-bench?

The operator-trivy module in Deckhouse Platform (DP) and kube-bench check the cluster against the same CIS Kubernetes Benchmark, but they run the checks differently and report results in different formats.

What is kube-bench?

kube-bench is a separate utility from Aqua Security that checks Kubernetes cluster compliance with CIS Benchmark requirements. It runs shell commands on nodes to analyze configuration.

To run kube-bench checks in a DP cluster yourself, follow the instructions for running kube-bench in a cluster in the deckhouse module FAQ.

How does the module implement CIS checks?

The module runs CIS checks with Trivy Operator, which:

• Runs as a Kubernetes operator inside the cluster
• Checks objects in all namespaces except the ones excluded from configuration scanning
• Stores results in ClusterComplianceReport
• Exports metrics to Prometheus

Identifier mapping

The table shows which checks cover each CIS section in kube-bench and in DP:

How to compare results?

Use the CIS Control ID (for example, 1.2.1 or 5.2.2) to match results. When kube-bench runs the CIS 1.12 benchmark (--benchmark cis-1.12), the IDs are the same in both tools.

Example: If kube-bench shows [FAIL] 1.2.16 Ensure that the --audit-log-path argument is set (Automated), find the same control in DP:

d8 k get clustercompliancereports.trivy.deckhouse.io cis -ojson | jq '.status.detailReport.results | map(select(.id == "1.2.16"))'

More details are in CIS Kubernetes Benchmark checks.

Why do some CIS checks always show PASS for system components?

Some checks are disabled for system namespaces (kube-system and d8-*) because system components require elevated privileges.

The full list of disabled checks is in CIS Kubernetes Benchmark checks.

Why are checks 5.1.2 and 5.1.3 not shown in metrics?

Checks 5.1.2 (Minimize access to secrets) and 5.1.3 (Minimize wildcard use in Roles and ClusterRoles) are excluded from metrics due to a high number of false positives for system components.

The checks are still performed, and results are available in the ClusterComplianceReport resource:

d8 k get clustercompliancereports.trivy.deckhouse.io cis -ojson | jq '.status.detailReport.results | map(select(.id == "5.1.2" or .id == "5.1.3"))'

What CIS Benchmark version is used?

The module implements checks according to CIS Kubernetes Benchmark v1.12 specification.

Versions of the components used by the module:

How often are CIS checks performed?

CIS Benchmark checks run on the schedule set by scanPeriods.complianceRescanCron (default 0 */6 * * *, every 6 hours) and on operator startup.

How to limit scan job load

Scan jobs that pull and analyze container images can be resource-intensive. Use the following module settings:

• concurrentScanJobsLimit: Maximum number of scan jobs that run at the same time (default 10).
• scanJobResources: CPU and memory requests/limits for scan job pods. If omitted, the module uses 100m/100M requests and 500m/500M limits.
• nodeSelector and tolerations: Where the operator and scan jobs are scheduled.

Where are scan reports stored?

The storage of scan reports is set by the securityStorageForReports parameter:

• true (default): reports are served by the security-storage aggregated API server under the trivy.deckhouse.io API group and are stored on the security-storage PVC;
• false: reports are stored in etcd as custom resources of the aquasecurity.github.io API group. On a node with many packages a report can exceed the 3 MB object size limit of etcd, so keep the severities filter.

The aquasecurity.github.io CRDs are installed in both modes. In d8 k commands, specify the resource together with the API group of the current mode, for example clustercompliancereports.aquasecurity.github.io if securityStorageForReports is false. The commands in this documentation use the default group.

When the parameter value changes, existing reports are not moved. The operator rescans workloads and nodes and recreates the reports in the new storage, so the reports are incomplete until the rescan finishes. What happens to the reports of the previous mode depends on the direction of the switch:

• after switching to false, the security-storage PVC is deleted together with the reports it holds, unless nodeAgent.enabled keeps security-storage running;
• after switching back to true, the reports in etcd remain in the cluster and are no longer updated, except the ClusterComplianceReport resources, which the module deletes.

If nodeAgent.enabled kept security-storage running in the false mode, a switch back to true returns the ClusterComplianceReport resources of trivy.deckhouse.io with the status of the previous period. The operator rebuilds such a report at once when its status was built before the current interval of scanPeriods.complianceRescanCron, otherwise at the next run of that schedule.

To delete the reports left in etcd after switching back to true, run the following command:

for r in $(d8 k api-resources --api-group=aquasecurity.github.io -o name); do d8 k delete "$r" --all -A; done

The command deletes every resource of the aquasecurity.github.io group in all namespaces. The CRDs themselves remain.

How to expand the module’s disk (PVC)

The module stores data on the following PVCs in the d8-operator-trivy namespace:

• security-storage: Runtime profile data (ApplicationProfile, NetworkNeighborhood) and, unless securityStorageForReports is false, scan reports (VulnerabilityReport, SbomReport, ClusterComplianceReport). The size is set by securityStoragePVCRequest (default 10 GiB).
• data-trivy-server-0: The Trivy vulnerability database cache, 5Gi.
• data-trivy-provider-0: The trivy-provider cache, 2Gi. Created only when denyVulnerableImages.enabled is true and the admission-policy-engine module is enabled.
• data-registry-scanner-0: The registry-scanner cache, 2Gi. Created only when registryScanning.enabled is true.

The sizes of the data-* PVCs are set in the module templates and are not configurable. If apiServerStorageClass (for security-storage) or storageClass (for the data-* PVCs) is set to false, the module uses emptyDir instead of the corresponding PVCs.

A PVC can be expanded only if its StorageClass supports volume expansion. A PVC cannot be shrunk. To expand a PVC, follow these steps:

1. Make sure the StorageClass allows expansion (allowVolumeExpansion: true):

   d8 k get storageclass <STORAGE_CLASS> -o jsonpath='{.allowVolumeExpansion}'

   <STORAGE_CLASS> is the StorageClass used by the PVC (the STORAGECLASS column in the output of d8 k -n d8-operator-trivy get pvc).

2. Increase the PVC size:

   • For security-storage, increase securityStoragePVCRequest. The module renders the PVC from this parameter, so do not patch this PVC manually. If the PVC already requests more than the parameter, the module keeps its size. If the StorageClass does not allow expansion, the module keeps the size of a smaller PVC as well.

   • For the data-* PVCs, increase the storage request on the existing PVC:

     d8 k -n d8-operator-trivy patch pvc <PVC_NAME> --type merge -p '{"spec":{"resources":{"requests":{"storage":"<NEW_SIZE>"}}}}'

     Where:

     • <PVC_NAME>: data-trivy-server-0, data-trivy-provider-0, or data-registry-scanner-0.
     • <NEW_SIZE>: New size, larger than the current one, for example 10Gi.

     The volumeClaimTemplates field of a StatefulSet is immutable, so its template keeps the original size. The StatefulSet keeps using the existing expanded PVC. If the PVC is recreated, for example after a StorageClass change, it gets the size from the template again.

3. Depending on the CSI driver, the volume expands online or after the consuming pod is restarted.