The operator-trivy module: vulnerability scanning

In a Deckhouse Platform (DP) cluster, vulnerability scanning is performed by Trivy Operator. It generates a Software Bill of Materials (SBOM) for workloads in namespaces labeled security-scanning.deckhouse.io/enabled and scans them for vulnerabilities. If no namespace has the label, the default namespace is scanned.

Supported object types for scanning

The following object types are supported:

• Container images stored in a registry.
• Node host filesystems in the cluster.

Analysis architecture

Regardless of object type, scanning consists of the following stages:

Vulnerability scanning mechanism

The vulnerability scanner is Trivy’s core component: it detects known CVEs using vulnerability databases. The operator-trivy module scans OS packages and components (binaries, libraries) and application dependencies (libraries, modules, and other dependencies).

The operator automatically triggers scans when new or changed workloads appear in namespaces labeled security-scanning.deckhouse.io/enabled. Reports are regenerated when they become older than scanPeriods.workloadRescanPeriod (default 24h). Set the parameter to "" to disable periodic rescanning.

Operating system packages

The scanner detects the OS distribution of the target and matches its packages against the vulnerability database for that distribution. For the sources of the data, see Vulnerability data sources.

Application dependencies

In container images, the scanner finds application dependencies by the metadata of installed packages and by binaries. Lockfiles and build manifests, such as package-lock.json, requirements.txt or pom.xml, are not analyzed in images. Per ecosystem, the scanner uses specialized vulnerability databases, primarily GitLab Advisory Database and GitHub Advisory Database.

The following files are analyzed:

Scan results and the VulnerabilityReport resource

Vulnerability scan results are stored in the cluster as VulnerabilityReport objects of the trivy.deckhouse.io group (aquasecurity.github.io if securityStorageForReports is false). Each report is created in the workload namespace and named <KIND>-<WORKLOAD_NAME>-<CONTAINER_NAME>, where <KIND> is the workload kind in lowercase. The report lists discovered CVEs with severity, fix status, affected packages, and references.

The report is tied to the parent object via ownerReferences: when the image changes or the workload is removed, the stale VulnerabilityReport is garbage-collected and recreated if needed.

By default the report includes all severity levels. Use the severities parameter to keep only selected levels in the report. Additional fields from the vulnerability database can be added with additionalVulnerabilityReportFields. To copy labels from the scanned workload onto the report, use reportResourceLabels.

Vulnerability trend metrics

The module records aggregated vulnerability series in Prometheus. Deckhouse Console builds the vulnerability trend charts from them. The series are calculated from the trivy_vulnerability_id metric of the operator, so they cover the namespaces where image scanning is enabled.

The following series are available:

• trivy:workload_vulnerabilities:count — number of unique vulnerabilities of a controller, with the namespace, controller_kind, controller_name and severity labels;
• trivy:workload_vulnerability_id:present — vulnerability identifiers present in a controller, with the namespace, controller_kind, controller_name, severity and vuln_id labels; the value is always 1.

A vulnerability is counted once per controller, however many containers, images or packages of the controller carry it. When the packages give it different severities, the series keep the highest one; an empty severity is recorded as Unknown. The other series of the trivy:workload_ family are intermediate.

The report of a Deployment belongs to its ReplicaSet, which changes with every rollout. The series map the ReplicaSet to its Deployment with the kube_replicaset_owner metric, so a Deployment keeps one history across revisions.

A report is deleted when it becomes older than scanPeriods.workloadRescanPeriod and is absent until the workload is scanned again. On a large cluster the rescan takes hours, so while a workload has no report, the series keep the vulnerabilities of its last report for six hours. Without that, every rescan would show up as fixed and then new vulnerabilities. A gap longer than six hours, for example when the scan keeps failing, still shows up this way. The window is checked per workload, not per container. If one container of a workload has a report and another has none, the vulnerabilities of the other container are absent until it is scanned again. One scan job reports all containers of a workload, so their reports usually expire together.

The window applies only while a workload has no report, so the series follow the workload itself:

• as soon as a report exists, the series hold exactly its vulnerabilities: an image update, a VEX statement or a vulnerability database update changes them at once, and a report without vulnerabilities clears them;
• a deleted workload leaves the series within ten minutes, once kube-state-metrics stops describing it;
• a Deployment counts only its current revision, the ReplicaSet that ran last: while a rollout or a rollback runs, both revisions run and both count; the previous revision leaves the series as soon as its ReplicaSet is scaled down to zero, even if its report outlives the rollout;
• a Deployment scaled down to zero keeps the vulnerabilities of its current revision until its report expires after scanPeriods.workloadRescanPeriod, and six hours longer: the operator does not rescan a ReplicaSet without pods.

The following query returns the number of critical vulnerabilities in a namespace:

sum(trivy:workload_vulnerabilities:count{namespace="<NAMESPACE>", severity="Critical"})

The following query returns the vulnerabilities that a controller has now and did not have seven days ago:

trivy:workload_vulnerability_id:present{namespace="<NAMESPACE>", controller_kind="<KIND>", controller_name="<NAME>"}
  unless trivy:workload_vulnerability_id:present{namespace="<NAMESPACE>", controller_kind="<KIND>", controller_name="<NAME>"} offset 7d

The series carry the job="operator-trivy" label, so the longterm Prometheus federates them as well. Each Prometheus keeps the history for its own retention period: the main one for 15 days by default, the longterm one longer.

SBOM generation and processing

The generated SBOM is stored in the cluster as an SbomReport object of the same group. Each report is created in the workload namespace and includes the following data:

• Container image metadata (repository, tag, digest)
• Component inventory in CycloneDX format: OS packages and application dependencies with PURLs, licenses, and supplier information
• Dependency graph between components
• Summaries of component and dependency counts

The operator watches workloads in the cluster. When a workload is created or updated, a scan job runs and produces both SbomReport and VulnerabilityReport. Both follow the <KIND>-<WORKLOAD_NAME>-<CONTAINER_NAME> naming scheme and are tied to the parent object via ownerReferences: when the workload is deleted, Kubernetes garbage collection removes the reports.

SbomReport and VulnerabilityReport complement each other: VulnerabilityReport lists CVEs, while SbomReport captures the full software inventory.

To disable SBOM generation, set disableSBOMGeneration to true. When this option is enabled, existing SbomReport objects are deleted from the cluster once.

Node scanning

The module can scan the host filesystem of each Kubernetes node for OS package vulnerabilities and, optionally, secrets stored in files on the node. Results are stored in NodeVulnerabilityReport objects, a cluster-wide resource.

The feature is disabled by default. Enable it in ModuleConfig operator-trivy:

spec:
  settings:
    nodeScanning:
      enabled: true

Node scanning is controlled by the nodeScanning setting:

Example configuration:

spec:
  settings:
    nodeScanning:
      enabled: true
      scanners:
        - Vuln
      pkgTypes:
        - OS
      severities:
        - CRITICAL
        - HIGH
      concurrentLimit: 1

Node scan job resources

Each node is scanned by a scan-noderootfs-* job that runs on that node. Its requests and limits are set by nodeScanning.resources; each omitted field keeps its default value: 100m/256Mi requests and 1/2Gi limits.

Values that do not match the needs of the scan affect it as follows:

• Memory limit. A scan of the entire node filesystem can take up to 2 GiB of memory. A scan job that exceeds the limit is terminated (OOMKilled), and the node gets no NodeVulnerabilityReport. A lower limit does not reduce the memory the scan needs, so every attempt fails the same way.
• CPU limit. A lower limit slows the scan down but does not break it. The job is stopped (DeadlineExceeded) when it runs longer than nodeScanning.timeout, which defaults to the scan job timeout of 5m. Raise the timeout together with a low CPU limit.
• Requests. The job is bound to its node, so the scheduler does not place it: the kubelet of the node admits it if the requests fit the free allocatable resources of the node. A request that does not fit fails the job with OutOfmemory or OutOfcpu. A small request lets the job start on a node that has less free memory than the scan takes.

A default request above the limit you set is lowered to that limit. A ModuleConfig with a request above the limit, explicit or default, is rejected when you save it. Only a change of nodeScanning.resources is checked, so a ModuleConfig saved with such values earlier still accepts other changes. For example, requests.memory: 3Gi without limits.memory is rejected, because the default limit is 2Gi.

To skip nodes that cannot spare this memory, label the suitable nodes and list the label in nodeSelector:

spec:
  settings:
    nodeScanning:
      enabled: true
      nodeSelector:
        <LABEL_KEY>: "<LABEL_VALUE>"
      resources:
        limits:
          memory: 3Gi

Only nodes with the label <LABEL_KEY>: <LABEL_VALUE> are scanned, and each scan job may use up to 3 GiB of memory.

Failed node scans

A node scan that ends without a report is repeated five minutes later. After three failed attempts in a row the operator stops scanning the node, and the NodeVulnerabilityScanFailed alert fires. The alert names the node and the reason for the last failure, for example OOMKilled, DeadlineExceeded, OutOfmemory or JobRejected (the API server or an admission policy rejected the scan job). An unavailable API server or admission webhook does not use up attempts: the operator retries such errors on its own. A scan job denied by an admission webhook, such as the policy engine of the cluster, counts as JobRejected, but the operator keeps retrying it every five minutes after the third attempt. The alert fires while the denial lasts and resolves once the webhook admits the job.

The operator keeps the count in memory and starts it again in the following cases:

• the operator restarts, for example after a change of nodeScanning or of another setting that the operator reads;
• the rescan annotation of the node gets a new value;
• the node gets its report.

A restart also clears the alert until the node fails three more times. To scan the node again without changing the settings, set the rescan annotation:

d8 k annotate node <NODE_NAME> trivy-operator.aquasecurity.github.io/node-rootfs-scan="$(date +%s)" --overwrite

The node gets three new attempts, and the alert resolves.

Registry scanning

In addition to scanning images running in the cluster, the module supports periodic scanning of image repositories stored in external container registries.

Enable the feature in ModuleConfig operator-trivy:

spec:
  settings:
    registryScanning:
      enabled: true
      concurrentScans: 5

The registryScanning.concurrentScans parameter limits how many image scans run in parallel across all targets. The default is 5.

Setting up registry repositories scanning

A scan target is described by a RegistryScanTarget, a cluster-wide resource:

apiVersion: deckhouse.io/v1alpha1
kind: RegistryScanTarget
metadata:
  name: preprod-myapp
spec:
  registry: preprod.registry.example.com
  repositories:
    - myapp/backend
    - myapp/frontend
  rescanPeriod: 1d
  # Optional. Scan only tags matching the regular expression.
  tagFilter: "^v[0-9]+"
  # Optional. Scan at most 50 tags per repository (default 100).
  maxTags: 50
  # Optional. Credentials for a private registry.
  registrySecretRef:
    name: image-pull-credentials-secret
    namespace: myapp

If the registry requires a CA certificate that is not in the system trust store, add it via additionalRegistryCA.

Scan results

Results are stored in RegistryImageVulnerabilityReport objects, a cluster-wide resource, one per scanned image tag. Each report is owned by its RegistryScanTarget. When a RegistryScanTarget is deleted, Kubernetes garbage collection removes all its reports.

To list all registry scan reports, run:

d8 k get registryimagevulnerabilityreport

To inspect a specific report, run:

d8 k describe registryimagevulnerabilityreport <REPORT_NAME>

Replace <REPORT_NAME> with the name of a report from the list.

A report is stored in etcd, so its size is limited. When the report of an image exceeds 1 MiB, the scanner drops data from it in the following order until it fits:

1. The links lists; primaryLink stays.
2. The descriptions of the vulnerabilities.
3. The CVSS vectors; score stays.
4. The vulnerabilities of the lowest severities.

The summary counters always describe the complete scan result. The registry-scanner.deckhouse.io/trimmed annotation of the report lists the dropped data, for example links,description or links,description,cvss,vulnerabilities=1200/3400.

The report is re-created after the interval specified by rescanPeriod elapses. To force an immediate rescan, remove the registry-scanner.deckhouse.io/last-scan-time annotation from the RegistryScanTarget. The controller starts a new scan cycle at once, without waiting for the next scheduled run.

The scanner starts a cycle ahead of that interval in the following cases:

• The spec of the RegistryScanTarget changed. An added repository or a corrected tag filter is scanned at once.
• The previous cycle failed before it scanned anything, for example because the tag filter is not a valid regular expression. The retry runs in five minutes.
• The previous cycle scanned only part of its images. The retry runs in an hour, and the wait grows while the target keeps failing, up to rescanPeriod.

The first cycle after the component starts waits for the Java index database to be downloaded. Until then a target that has never been scanned reports Ready=False with the WaitingForJavaDB reason, and a target that already has a result keeps it.

Checking the scan state

The RegistryScanTarget status shows the time of the last scan cycle and its result.

To list the RegistryScanTarget objects, run:

d8 k get registryscantarget

To view the status of one of them, for example my-registry, run:

d8 k describe registryscantarget my-registry

The Ready condition appears when the first cycle of a target starts, or earlier with the WaitingForJavaDB reason while the Java index database is being downloaded. It is True after a successful cycle and False in every other state, and the reason field tells the states apart:

While the Java index database is being downloaded, no target is scanned. The component reports itself as not ready until the download succeeds, so watch the readiness of its pod when cycles stop happening.

When tagFilter is set, status.targets lists the matched tags of at most 50 repositories.

A report deleted by hand is re-created by the next cycle of its target, not immediately. To re-create it at once, force a rescan.

Reports without the registry-scanner.deckhouse.io/target label are removed by the first cycle that reads every repository of its target and scans every image of it. A target whose catalog stays longer than maxRepositories, or that always has a repository it cannot read or an image it cannot scan, keeps them until that changes, and its Ready condition says so. A repository that answers with no tags does not hold the cleanup back: an empty list is a complete answer.

Reports are removed when their image leaves the target: its tag is gone from the repository, or the repository is no longer selected. The following reports are kept:

• Reports of a repository the cycle could not read: one that answered with an error, one whose tag list the registry did not serve whole, and one left out by maxRepositories. A repository whose tag list came in part is scanned as far as the registry showed it, and the Ready condition names how many repositories that happened to.
• Reports of a repository that answered with no tags. They are kept until the repository leaves the target.

Tags beyond maxTags are left out by choice, so their reports are removed.

Blocking vulnerable containers

The module enforces policy on running containers that use vulnerable images at Kubernetes admission time. It combines OPA Gatekeeper (the admission-policy-engine module) and Trivy: Trivy acts as a data provider for Gatekeeper with up-to-date scan results, and a Rego policy admits or denies the object based on that data.

The constraint applies when Pods, Deployments, StatefulSets, and DaemonSets are created or updated in namespaces labeled security.deckhouse.io/trivy-provider: "".

Configuration

Blocking is controlled by the denyVulnerableImages module setting:

Configuration example

spec:
  settings:
    denyVulnerableImages:
      enabled: true
      allowedSeverityLevels:
        - UNKNOWN
        - LOW
        - MEDIUM
      registrySecrets:
        - name: my-private-registry-pull-secret
          namespace: my-app

In this example, images with HIGH or CRITICAL vulnerabilities are blocked. Images with only UNKNOWN, LOW, or MEDIUM findings are allowed. The my-private-registry-pull-secret Secret is used so that trivy-provider can pull images from that registry during admission checks.

Suppressing vulnerabilities with VEX attestations

VEX (Vulnerability Exploitability eXchange) is a machine-readable statement that declares whether a known CVE is actually exploitable in a specific product version. A VEX statement can say, for example, that a vulnerability exists in a library the image ships but is not reachable because the affected code path is never called.

When useVEXFromOCI is enabled, Trivy fetches VEX attestations that are attached to scanned images in the OCI registry (as OCI referrers) and uses them to filter CVE findings. A vulnerability is omitted from the resulting VulnerabilityReport or RegistryImageVulnerabilityReport if a VEX statement marks it as not_affected or fixed for that image.

This applies to workload scanning (images running in the cluster), registry scanning (RegistryScanTarget), and admission-time checks (denyVulnerableImages), where a suppressed vulnerability does not block the image.

Requirements

• The registry must support the OCI Referrers API (OCI Distribution Spec v1.1+). Most modern registries (Harbor, Quay, GHCR, ECR) support it.
• VEX attestations must be stored as OCI referrers of the image.
• If the registry certificate is issued by a private certificate authority (CA), add the CA certificate to additionalRegistryCA. Trivy fetches the attestations with the same certificates as the image itself.
• The registry must not be listed in insecureRegistries and, for registry scanning, must not have insecure: true in the RegistryScanTarget. Trivy fetches attestations only over HTTPS with certificate verification, so an image from such a registry is scanned without VEX attestations.

Enabling

Add the following option to ModuleConfig operator-trivy:

spec:
  settings:
    useVEXFromOCI: true

Mapping CVE identifiers to BDU

When linkCVEtoBDU is enabled, the module replaces CVE identifiers in VulnerabilityReport objects with identifiers from the FSTEC of Russia threat database (BDU), for example BDU-2021-01234. Only the vulnerabilityID field of a vulnerability is replaced. A CVE without a matching BDU entry keeps its CVE identifier. NodeVulnerabilityReport and RegistryImageVulnerabilityReport objects are not changed.

To enable the mapping, set the parameter in ModuleConfig operator-trivy:

spec:
  settings:
    linkCVEtoBDU: true

Compliance reports

The module deploys ClusterComplianceReport objects for selected compliance frameworks. Each report is regenerated according to scanPeriods.complianceRescanCron (default 0 */6 * * *, every 6 hours) and maps controls of a framework to the underlying configuration and workload checks performed by Trivy.

The set of deployed reports is controlled by the complianceReports.enabled parameter.

Configuration example

The following ModuleConfig operator-trivy fragment deploys four of the reports:

spec:
  settings:
    complianceReports:
      enabled:
        - CIS
        - NSA
        - FSTEC-21
        - FZ-187

To list the compliance reports, run:

d8 k get clustercompliancereports

To inspect a report, for example fstec-21, run:

d8 k describe clustercompliancereport fstec-21

What compliance reports cover

By default, complianceReports.skipSystemResources is enabled, so compliance reports focus on workloads that the cluster owner can fix. The module excludes the following objects from compliance reports:

• Objects in namespaces matching d8-*, kube-* and default
• Cluster-wide objects labeled heritage=deckhouse, which belong to DP and its modules
• Built-in RBAC objects (kubernetes.io/bootstrapping=rbac-defaults)

The control plane itself is still checked against the relevant infrastructure controls (for example, CIS Kubernetes Benchmark section 1.x for kube-apiserver, etcd, kube-scheduler settings). Only generic workload-level findings about platform pods are filtered out.

Vulnerability scanning is independent of these filters and keeps reporting CVEs and secrets for every scanned image, including platform ones.

To exclude an additional namespace from compliance reports, for example, an experimental or short-lived environment, label the namespace with security.deckhouse.io/skip-compliance=true. The label works regardless of the complianceReports.skipSystemResources value:

d8 k label namespace my-namespace security.deckhouse.io/skip-compliance=true

Security events

When the security-events-manager module is enabled, the module sends it a security event every time a scan saves a report with findings. The module ships the SecurityEventDefinition resources and the operator-trivy PodSecurityEventShipper resource in the d8-operator-trivy namespace, so no additional configuration is required.

Events are built from the logs of three components:

• operator — reports on workloads, nodes, configuration audit, RBAC, infrastructure and compliance;
• registry-scanner — reports on images from the registries listed in RegistryScanTarget resources;
• trivy-provider — images rejected by the policy described in the “Blocking vulnerable containers” section.

Event list

The table lists the events the module produces. A report produces one event, whatever the number of its findings.

The node and registry events are produced only when node scanning and registry scanning are enabled, as described in the “Node scanning” and “Registry scanning” sections.

Event severity

An event marked “By findings” takes the severity of the most severe finding in the report: CRITICAL becomes Critical, HIGH becomes High, MEDIUM becomes Medium, LOW and UNKNOWN become Low.

The securityEvents.minimumSeverity parameter sets the lowest severity that produces an event. With the default value HIGH, a report produces an event only when it contains CRITICAL or HIGH findings. Rejected images, registry scan failures and failed compliance controls always produce an event.

To receive events about reports with MEDIUM and LOW findings too, set the parameter to LOW:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: operator-trivy
spec:
  version: 1
  settings:
    securityEvents:
      minimumSeverity: LOW

Nearly every image contains MEDIUM or LOW vulnerabilities, so with this value an event is produced for almost every scanned workload. The parameter discards the findings before they leave the cluster node, while the defaultSeverityThreshold parameter of ClusterSecurityEventConfig filters the events already collected, separately for each destination.

Event contents

An event describes the scanned object rather than the report. For a workload, object holds the kind, the name and the namespace of the workload, for example a ReplicaSet; for a node, the node name; for a registry image or a rejected image, the image reference.

The metadata.extra field carries the rest of the data:

• critical, high, medium, low, unknown and total — the number of findings of each severity;
• reportKind, reportName and reportNamespace — the report that lists the findings, with reportNamespace only for a report that lives in a namespace;
• container, image and imageDigest — the container and the image of a workload;
• component — the module component that produced the event.

The list of vulnerabilities is not copied into the event. To view it, open the report named in reportName.

Event frequency

An event is produced every time the report is saved, not only when new findings appear. Workload reports are regenerated every scanPeriods.workloadRescanPeriod, so a vulnerable workload produces an event at least once in that period.

D8_TRIVY_REGISTRY_SCAN_FAILED follows the state of a RegistryScanTarget instead. registry-scanner retries a failed target every few minutes, but the event is produced only when the target enters the failed state or the reason or the message of its Ready condition changes, so a target that keeps failing the same way produces one event.

The source of the events is podSecurityEventShipper/d8-operator-trivy/operator-trivy/operator-trivy. To stop receiving the events, exclude this source from the enabledSources or enabledSourcesMasks parameter of the ClusterSecurityEventConfig resource.

Security score

The module aggregates the security state of the cluster into a single score from 0 to 100 and calculates the same score for every user namespace. Both values are published as objects and shown in Deckhouse Console.

The cluster score is stored in the ClusterSecurityScore cluster, and a namespace score is stored in the SecurityScore security-score in that namespace. In both, the data is in the score block.

The split exists for access control: a namespace score is granted together with the namespace itself, by the d8:use:capability:module:operator-trivy:view role, while the cluster score is granted by the d8:manage:permission:module:operator-trivy:view role. A project user sees the scores of their own namespaces without also receiving a map of the weak points of the whole cluster.

The score is recalculated when a Namespace or a NetworkPolicy changes, when ModuleConfig admission-policy-engine, operator-trivy or runtime-audit-engine changes, and on a 10-minute schedule. Reports are read on every recalculation, so all components reflect the current state of the cluster.

Changes to namespaces labeled heritage=deckhouse or heritage=upmeter do not trigger a recalculation. Changes to the other excluded namespaces, those matching d8-*, kube-* and default and those labeled security.deckhouse.io/skip-compliance=true, trigger a recalculation, but these namespaces are still left out of the score.

Score components

The cluster score is a weighted average of seven components, each normalized to a value from 0 to 1. The weights add up to 100, so a cluster whose every component equals 1 scores exactly 100.

Only user namespaces are counted. The module excludes namespaces matching d8-*, kube-* and default, namespaces labeled heritage=deckhouse or heritage=upmeter, and namespaces labeled security.deckhouse.io/skip-compliance=true.

Not every report kind affects the score. RbacAssessmentReport and InfraAssessmentReport are not part of any component: they are used in the compliance reports and reach the score only through the compliance component. This is why control plane checks in kube-system, which the module keeps producing, do not move the misconfig component.

Excluding a namespace from the score does not exclude it from scanning. Configuration scanners run cluster-wide, and the vulnerability scanner covers every namespace labeled security-scanning.deckhouse.io/enabled, including a system one. The exclusion controls only which namespaces the score is calculated over.

Components based on findings

The cve, misconfig and secrets components convert finding counts into a severity load, and the load into a sub-score by the formula 1 / (1 + load / k). The sub-score approaches 0 as findings accumulate but never reaches it, so no single component can zero out the whole score.

The load and the constant k differ per component:

Findings of low and unknown severity do not affect the score. Exposed secrets are scored the most aggressively: a single critical finding already halves the secrets component.

Pod security component

The psa component combines the strictness of the policy applied to a namespace with the way admission-policy-engine enforces it. The policy is taken from the security.deckhouse.io/pod-policy label of the namespace. When the label is not set, the policy is taken from the podSecurityStandards.defaultPolicy parameter of the admission-policy-engine module if the module is enabled, and is Privileged otherwise.

Each policy has the following strictness:

• Restricted: 1
• Baseline: 0.6
• Privileged: 0

The result is multiplied by the coefficient of the enforcement mode:

• Deny: 1
• Warn: 0.5
• Dryrun: 0.2
• admission-policy-engine disabled: 0.1

A Restricted policy in Dryrun mode therefore scores far lower than the same policy in Deny mode. The final and the pre-multiplier values are published in the value and raw fields of the component itself, while the enforcement mode and its coefficient are published once, in the modules.admissionPolicyEngine block.

Namespace score

The namespace score uses the same components, weights and constants as the cluster score, except for compliance: compliance reports are cluster-scoped, so their result cannot be attributed to a namespace. Node vulnerabilities are excluded for the same reason. Because the constants are shared, a namespace that holds all of the findings in the cluster ends up with the same component values as the cluster itself.

The compliance component means the cluster score and the namespace average in the average field are calculated over different sets of components. Never compare those two values with each other, even when every shared component matches.

The weighted average is renormalized over the weights actually used, so a namespace whose every applicable component equals 1 also scores 100.

A component based on reports carries the notMeasured field when no report of the corresponding type exists. The rule is the same for the cluster and for a namespace: without reports, the component is excluded from the calculation instead of counting as a clean result. A cluster where nothing is scanned therefore does not collect the full cve and secrets weight for an absence of findings.

The compliance component is an exception to that rule: with no compliance report deployed, it scores 1 and keeps its weight. Compliance reports are enabled on demand through the complianceReports.enabled parameter, so their absence is a configuration choice rather than an unobserved area. This is one more reason not to compare the cluster score with the namespace average.

A namespace that is not labeled for scanning and has no reports is limited to 60: coverage is 0, and only the psa and network weights remain.

An unscanned namespace usually has reports anyway. ConfigAuditReport objects are created regardless of the security-scanning.deckhouse.io/enabled label, so an unscanned namespace usually has a known misconfig component and a larger set of applicable weights. What such a namespace always loses is the entire coverage weight, which alone puts a score of 100 out of reach.

The same rule produces the opposite case. A namespace that is labeled for scanning, carries a NetworkPolicy and applies the Restricted policy, but has no report yet, scores 100: the three report-backed components are excluded from the calculation, and the remaining ones are all at their maximum. Right after scanning is enabled, and until the first reports arrive, such a namespace therefore reads as perfect.

Read the score together with the notMeasured field of the components: a value of 100 with cve, misconfig or secrets marked as not measured means the namespace is configured well and nothing has been checked yet, not that the namespace is clean.

Namespace scores are stored in objects of their own, so their number is not capped. The score.namespaceScores block of ClusterSecurityScore holds a summary only: total is the number of scored namespaces and average is their average score. For interfaces that are not allowed to list SecurityScore objects across the cluster, ClusterSecurityScore also keeps the score.topNamespaces block with the namespaces that need attention first.

SecurityScore is deleted together with its namespace. When a namespace stops being scored, for example, after it gets the security.deckhouse.io/skip-compliance=true label, the module deletes its SecurityScore.

Viewing the score

To list the scores of every namespace you have access to, run:

d8 k get securityscores.deckhouse.io --all-namespaces

To get the score of a single namespace with all its components, run:

d8 k -n <NAMESPACE> get securityscores.deckhouse.io security-score -o yaml

To get the cluster score, run:

d8 k get clustersecurityscores.deckhouse.io cluster -o yaml

Replace <NAMESPACE> with the name of the namespace.

The output of the first command lists the score, the scanning state and the applied pod security standard for every namespace. The full objects carry the value of every component and the findings it was calculated from.

The commands use the full resource names with the group. A name without the group, including the short names secscore and clustersecscore, may match a resource of another module.

Score metrics

The module exports the cluster score to Prometheus, so the value can be graphed and alerted on without reading ClusterSecurityScore.

The following metrics are available:

• deckhouse_trivy_security_score: Cluster score from 0 to 100
• deckhouse_trivy_security_score_component: Value of a single component from 0 to 1, with the component name in the component label
• deckhouse_trivy_security_namespace_score_average: Average score across all user namespaces
• deckhouse_trivy_security_namespaces_total: Number of user namespaces
• deckhouse_trivy_security_namespaces_scanned: Number of user namespaces with scanning enabled

A component that was not measured is not exported at all: the series for it disappears instead of reporting a value. Build an alert on deckhouse_trivy_security_score_component so that it covers that case: pair the condition with absent(), or rely on coverage, which is always present. An expression such as deckhouse_trivy_security_score_component{component="cve"} < 0.5 never fires on a cluster where nothing is scanned, which is where the alert matters most.

Per-namespace scores are not exported as metrics: on a large cluster that would add a time series for every namespace. Read them from SecurityScore objects instead.

Formula version

Weights, constants and calculation rules stay stable across releases. Any change to them raises the formulaVersion field published next to the score, both for the cluster and for namespaces. A change to a calculation rule raises it as well as a change to a weight.

Scores calculated with different formula versions are not comparable.

Working with private and insecure registries

By default the module expects all container registries to serve trusted TLS certificates. For environments that use self-signed certificates, plain HTTP, or that disable TLS verification, the following parameters are available.

Custom CA certificates

Use additionalRegistryCA to specify root certificates for private registries.

Each entry has a name (a human-readable name, for reference only) and ca (PEM-encoded certificate or chain). When specifying a chain, concatenate the certificates in PEM format with no extra blank lines between them.

spec:
  settings:
    additionalRegistryCA:
      - name: corporate CA
        ca: |
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----
      - name: CA with intermediate
        ca: |
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----

Insecure registries

Use insecureRegistries to allow the scanner to connect to specific registries over plain HTTP or HTTPS with unverified certificates. Each entry is a registry hostname (optionally with port).

spec:
  settings:
    insecureRegistries:
      - my.internal-registry.example.com
      - legacy-registry.example.com:5000

This setting applies to workload scanning and admission-time checks (denyVulnerableImages). For registry scanning, set insecure: true in the RegistryScanTarget.

Vulnerability data sources

The main vulnerability database, trivy-db, is the database of the Trivy project with additional sources added by DP. Every 6 hours DP takes the current Trivy database, adds its own sources to it, and publishes the result to the DP registry.

Sources from the Trivy database

The Trivy database is built by Aqua Security and includes the sources listed below. Aqua Security adds new sources to it; for the complete list, see data sources in the Trivy documentation.

OS distributions:

• AlmaLinux Product Errata
• Alpine SecDB
• Amazon Linux Security Center
• Azure Linux and CBL-Mariner Vulnerability Data
• Bottlerocket Security Advisories
• Chainguard Security Data
• Debian Security Tracker
• MinimOS Security Data
• Oracle Linux OVAL
• Photon OS CVE metadata
• Red Hat OVAL, also used for CentOS
• Rocky Linux updateinfo
• SUSE CVRF, for SUSE Linux Enterprise and openSUSE
• Ubuntu CVE Tracker
• Wolfi SecDB

Vendors of patched images and packages:

• Echo
• RapidFort Security Advisories
• Root.io Security Patches, for Alpine, Debian and Ubuntu
• Seal Security

Application dependencies:

• GitHub Advisory Database: Composer, Erlang, Go, Maven, npm, NuGet, pip, Pub, RubyGems, Rust, Swift
• GitLab Advisory Database Community
• Go Vulnerability Database
• Bitnami Vulnerability Database
• Julia Security Advisories
• Node.js Security Working Group
• PHP Security Advisories Database
• Ruby Advisory Database
• Aqua Security Vulnerability Database

Kubernetes vulnerabilities come from the Official Kubernetes CVE Feed.

Vulnerability details, such as the description and CVSS score, also come from the National Vulnerability Database (NVD).

Additional sources

DP adds the following sources to the Trivy database:

ALT Linux entries also carry the BDU identifiers that ALT publishes with its errata.

Database management

The module downloads the databases from the registry that DP is installed from and keeps them up to date on its own.

The Trivy server stores trivy-db in a 5 GiB PersistentVolumeClaim. If the storageClass parameter is false, or no StorageClass is set for the module or globally, the database is stored in an emptyDir and is downloaded again after every restart of the Trivy server.

The version of trivy-db and trivy-java-db the Trivy server uses is shown in the trivy-db-info ConfigMap:

d8 k -n d8-operator-trivy get configmap trivy-db-info -o yaml

The vulnerabilityDatabaseMaxAge parameter (default 168h) sets how old trivy-db may be before the TrivyVulnerabilityDatabaseOutdated alert fires. The related TrivyVulnerabilityDatabaseMissing alert fires when the trivy-db-info ConfigMap is absent or has no trivy-db.updatedAt timestamp for 30 minutes. Set the parameter to "" to disable both database freshness alerts.

The policies for configuration checks are not downloaded: they are built into the operator image and are updated together with the module. DP changes some of the upstream Trivy checks, see DP customizations.

Air-gapped environments

A cluster downloads the databases only from its own registry, so their freshness depends on how often they are copied there. Copy the databases with the d8 CLI utility every 6 hours, as often as new versions are published. For the commands, see Vulnerability scanner database update mechanism.

d8 mirror copies the databases under the same tags (trivy-db:2, trivy-java-db:1 and trivy-bdu:1), so no module configuration changes are needed. If the databases are not updated, scanning continues on the database already downloaded: vulnerabilities published after it are not detected, and once the database is older than vulnerabilityDatabaseMaxAge, the TrivyVulnerabilityDatabaseOutdated alert fires.