Vulnerability scanning

In a DKP cluster, vulnerability scanning is made with Trivy Operator. It automatically generates Software Bill Of Materials (SBOM) for all workloads and runs the scanner for specified objects without any manual actions required.

Supported object types for scanning

The following object types are supported:

• Container images stored in a registry.
• Node host filesystems in the cluster.

Analysis architecture

Regardless of object type, the scanning is made in three stages:

Vulnerability scanning mechanism

The vulnerability scanner is Trivy’s core component: it detects known CVEs using vulnerability databases. The operator-trivy module scans OS packages and components (binaries, libraries) and application dependencies (libraries, modules, and other deps).

The operator automatically triggers scans when new or changed workloads appear in namespaces where scanning is enabled. Reports are regenerated when they become older than scanPeriods.workloadRescanPeriod (default 24h). Set the parameter to "" to disable periodic rescanning.

Operating system packages

The scanner detects the target OS distribution and applies data from the CVE databases officially provided by Deckhouse. This improves detection accuracy and reduces false positives.

Data sources used to build the CVE databases are listed below:

• FSTEC of Russia threat database (BDU)
• AlmaLinux Errata
• Alpine SecDB
• ALTRepo Errata OVAL
• Amazon Linux Security Advisories
• Arch Linux Security Tracker
• Debian Security Tracker
• GitHub Security Advisory Database
• National Vulnerability Database (NVD)
• Oracle OVAL
• Photon Security Advisory
• Red Hat OVAL
• RED SOFT OVAL
• Rocky Linux UpdateInfo
• SUSE Security CVRF
• Ubuntu CVE Tracker
• Wolfi SecDB

Application dependencies

The scanner discovers and analyzes package manager manifests and lockfiles across more than twenty ecosystems. Per ecosystem it uses specialized vulnerability databases (primarily based on GitLab Advisory Database and GitHub Advisory Database).

Scan results and the VulnerabilityReport resource

Vulnerability scan results are stored in the cluster as the VulnerabilityReport custom resource in group trivy.deckhouse.io. Each report is created in the workload namespace, named <kind>-<workload-name>-<container-name>, and lists discovered CVEs with severity, fix status, affected packages, and references.

The report is tied to the parent resource via ownerReferences: when the image changes or the workload is removed, the stale VulnerabilityReport is garbage-collected and recreated if needed. This implements periodic rescanning of workloads.

By default the report includes all severity levels. Use the severities parameter to keep only selected levels in VulnerabilityReport. Additional fields from the vulnerability database can be added with additionalVulnerabilityReportFields. To copy labels from the scanned workload onto the report, use reportResourceLabels.

SBOM generation and processing

Generated SBOM is stored in the cluster as the SbomReport CRD in group trivy.deckhouse.io. Each report is created in the workload namespace and includes:

• container image metadata (repository, tag, digest);
• component inventory in CycloneDX format: OS packages and app dependencies with PURLs, licenses, and supplier information;
• a dependency graph between components;
• summaries of component and dependency counts.

The operator watches cluster resources. When a workload is created or updated, a scan job runs and produces both SbomReport and VulnerabilityReport. Both follow the <kind>-<workload-name>-<container-name> naming scheme and use ownerReferences — when the workload is deleted, reports are removed by Kubernetes garbage collection.

SbomReport and VulnerabilityReport complement each other: VulnerabilityReport lists CVEs, while SbomReport captures the full software inventory.

To disable SBOM generation, set disableSBOMGeneration to true. When this option is enabled, existing SbomReport resources are deleted from the cluster once.

Node scanning

The module can scan the host filesystem of each Kubernetes node for OS package vulnerabilities and, optionally, secrets stored in files on the node. Results are stored as cluster-wide NodeVulnerabilityReport resources.

The feature is disabled by default. Enable it in ModuleConfig operator-trivy:

spec:
  settings:
    nodeScanning:
      enabled: true

Node scanning is controlled by the nodeScanning setting:

Example configuration:

spec:
  settings:
    nodeScanning:
      enabled: true
      scanners:
        - Vuln
      pkgTypes:
        - OS
      severities:
        - CRITICAL
        - HIGH
      concurrentLimit: 1

Registry scanning

In addition to scanning images running in the cluster, the module supports periodic scanning of image repositories stored in external container registries.

Enable the feature in ModuleConfig/operator-trivy:

spec:
  settings:
    registryScanning:
      enabled: true
      concurrentScans: 5

The registryScanning.concurrentScans parameter limits how many image scans run in parallel across all targets. The default is 5.

Setting up registry repositories scanning

A scan target is declared as a cluster-scoped RegistryScanTarget resource:

apiVersion: deckhouse.io/v1alpha1
kind: RegistryScanTarget
metadata:
  name: preprod-myapp
spec:
  registry: preprod.registry.example.com
  repositories:
    - myapp/backend
    - myapp/frontend
  rescanPeriod: 1d
  tagFilter: "^v[0-9]+"   # Optional: Scan only tags matching the regular expression.
  maxTags: 50              # Optional: Scan at most 50 tags per repo (default 100).
  registrySecretRef:       # Optional: Credentials for a private registry.
    name: image-pull-credentials-secret
    namespace: myapp

If the registry requires a CA certificate that is not in the system trust store, add it via additionalRegistryCA.

Scan results

Results are stored as cluster-scoped RegistryImageVulnerabilityReport resources, one resource per scanned image tag. Each report is owned by its RegistryScanTarget. When a RegistryScanTarget is deleted, all its reports are garbage-collected automatically.

# List all registry scan reports.
d8 k get registryimagevulnerabilityreport

# Inspect a specific report.
d8 k describe registryimagevulnerabilityreport <name>

A report is stored in etcd, so its size is limited. When the report of an image exceeds 1 MiB, the scanner drops data from it in the following order until it fits:

1. The links lists; primaryLink stays.
2. The descriptions of the vulnerabilities.
3. The CVSS vectors; score stays.
4. The vulnerabilities of the lowest severities.

The summary counters always describe the complete scan result. The registry-scanner.deckhouse.io/trimmed annotation of the report lists the dropped data, for example links,description or links,description,cvss,vulnerabilities=1200/3400.

The report is re-created after the interval specified by rescanPeriod elapses. To force an immediate rescan, remove the registry-scanner.deckhouse.io/last-scan-time annotation from the RegistryScanTarget object. The controller will start a new scan cycle immediately, without waiting for the next scheduled run.

The scanner starts a cycle ahead of that interval in the following cases:

• The spec of the RegistryScanTarget changed. An added repository or a corrected tag filter is scanned at once.
• The previous cycle failed before it scanned anything, for example because the tag filter is not a valid regular expression. The retry runs in five minutes.
• The previous cycle scanned only part of its images. The retry runs in an hour, and the wait grows while the target keeps failing, up to rescanPeriod.

The first cycle after the component starts waits for the Java index database to be downloaded. Until then a target that has never been scanned reports Ready=False with the WaitingForJavaDB reason, and a target that already has a result keeps it.

Checking the scan state

The RegistryScanTarget status shows the time of the last scan cycle and its result:

d8 k get registryscantarget
d8 k describe registryscantarget my-registry

The Ready condition appears after the first cycle of a target. It is True after a successful cycle and False in every other state, and the reason field tells the states apart:

While the Java index database is being downloaded, no target is scanned. The component reports itself as not ready until the download succeeds, so watch the readiness of its pod when cycles stop happening.

When tagFilter is set, status.targets lists the matched tags of at most 50 repositories.

A report deleted by hand is re-created by the next cycle of its target, not immediately. To get it back at once, remove the registry-scanner.deckhouse.io/last-scan-time annotation from the RegistryScanTarget.

Reports written before the current release are removed by the first cycle that reads every repository of its target. A target whose catalog stays longer than maxRepositories, or that always has a repository it cannot read, keeps them until that changes, and its Ready condition says so. A repository that answers with no tags does not hold the cleanup back: an empty list is a complete answer.

Reports are removed when their image leaves the target: its tag is gone from the repository, or the repository is no longer selected. The reports of a repository the cycle could not read are kept — one that answered with an error, one that showed no tags, one whose tag list the registry did not serve whole, and one left out by maxRepositories. A repository whose tag list came in part is scanned as far as the registry showed it, and the Ready condition names how many repositories that happened to. Tags beyond maxTags are not protected this way: they are left out by choice, so their reports are removed.

Blocking vulnerable containers

The module enforces policy on running containers that use vulnerable images at Kubernetes admission time. It combines OPA Gatekeeper (the admission-policy-engine module) and Trivy: Trivy acts as a data provider for Gatekeeper with up-to-date scan results, and a Rego policy admits or denies the resource based on that data.

The constraint applies when Pod, Deployment, StatefulSet, and DaemonSet resources are created or updated in namespaces labeled security.deckhouse.io/trivy-provider: "".

Configuration

Blocking is controlled by the denyVulnerableImages module setting:

Configuration example

spec:
  settings:
    denyVulnerableImages:
      enabled: true
      allowedSeverityLevels:
        - UNKNOWN
        - LOW
        - MEDIUM
      registrySecrets:
        - name: my-private-registry-pull-secret
          namespace: my-app

In this example, images with HIGH or CRITICAL vulnerabilities are blocked. Images with only UNKNOWN, LOW, or MEDIUM findings are allowed. The my-private-registry-pull-secret Secret is used so that trivy-provider can pull images from that registry during admission checks.

Suppressing vulnerabilities with VEX attestations

VEX (Vulnerability Exploitability eXchange) is a machine-readable statement that declares whether a known CVE is actually exploitable in a specific product version. A VEX statement can say, for example, that a vulnerability exists in a library the image ships but is not reachable because the affected code path is never called.

When useVEXFromOCI is enabled, Trivy fetches VEX attestations that are attached to scanned images in the OCI registry (as OCI referrers) and uses them to filter CVE findings. A vulnerability is omitted from the resulting VulnerabilityReport or RegistryImageVulnerabilityReport if a VEX statement marks it as not_affected or fixed for that image.

This applies to both workload scanning (images running in the cluster) and registry scanning (RegistryScanTarget).

Requirements

• The registry must support the OCI Referrers API (OCI Distribution Spec v1.1+). Most modern registries (Harbor, Quay, GHCR, ECR) support it.
• VEX attestations must be stored as OCI referrers of the image.

Enabling

Add the following option to the ModuleConfig/operator-trivy

spec:
  settings:
    useVEXFromOCI: true

Mapping CVE identifiers to BDU

When linkCVEtoBDU is enabled, the module converts CVE records in vulnerability reports to identifiers from the FSTEC of Russia threat database (BDU).

spec:
  settings:
    linkCVEtoBDU: true

Compliance reports

The module deploys ClusterComplianceReport resources for selected compliance frameworks. Each report is regenerated according to scanPeriods.complianceRescanCron (default 0 */6 * * *, every 6 hours) and maps controls of a framework to the underlying configuration and workload checks performed by Trivy.

The set of deployed reports is controlled by the complianceReports.enabled parameter.

Configuration example

complianceReports:
  enabled:
    - CIS
    - NSA
    - FSTEC-21
    - FZ-187

In this example only four reports are deployed. To inspect a report, use:

d8 k get clustercompliancereports
d8 k describe clustercompliancereport fstec-21

What compliance reports cover

By default complianceReports.skipSystemResources is enabled, so compliance reports focus on workloads that the cluster owner can fix. The module automatically excludes from compliance:

• resources owned by Deckhouse and its modules;
• Kubernetes system components (built-in RBAC, control plane defaults, etc.).

The control plane itself is still checked against the relevant infrastructure controls (for example, CIS Kubernetes Benchmark section 1.x for kube-apiserver, etcd, kube-scheduler settings) — only generic workload-level findings about platform pods are filtered out.

Vulnerability scanning is independent of these filters and keeps reporting CVEs and secrets for every scanned image, including platform ones.

To exclude an additional namespace from compliance — for example, an experimental or short-lived environment — label the namespace:

d8 k label namespace my-namespace security.deckhouse.io/skip-compliance=true

Security score

The module aggregates the security state of the cluster into a single score from 0 to 100 and calculates the same score for every user namespace. Both values are published as resources and shown in the Deckhouse web interface.

The cluster score lives in a ClusterSecurityScore resource named cluster, and a namespace score lives in a SecurityScore resource named security-score in that namespace. In both resources the payload sits in the score block.

The split exists for access control: a namespace score is granted together with the namespace itself, by the d8:use:capability:module:operator-trivy:view role, while the cluster score is granted by the d8:manage:permission:module:operator-trivy:view role. A project user sees the scores of their own namespaces without also receiving a map of the weak points of the whole cluster.

The score is recalculated when a Namespace, a NetworkPolicy or a ModuleConfig changes, and on a 10-minute schedule. The report resources are read on every recalculation, so all components reflect the current state of the cluster.

Changes to namespaces labeled heritage=deckhouse or heritage=upmeter do not trigger a recalculation. Those namespaces are excluded from the score anyway, and upmeter creates and deletes a probe namespace every minute, which would otherwise recalculate the score twice a minute and re-read every report in the cluster for an unchanged result. The remaining excluded namespaces — those matching d8-*, kube-* and default, and those labeled security.deckhouse.io/skip-compliance=true — do trigger a recalculation, because a label selector cannot express those conditions. They are still left out of the score itself.

Score components

The cluster score is a weighted average of seven components, each normalized to a value from 0 to 1. The weights add up to 100, so a cluster whose every component equals 1 scores exactly 100.

Only user namespaces are counted. The module excludes namespaces matching d8-*, kube-* and default, namespaces labeled heritage=deckhouse or heritage=upmeter, and namespaces labeled security.deckhouse.io/skip-compliance=true.

Not every report kind affects the score. RbacAssessmentReport and InfraAssessmentReport are not part of any component: they feed the compliance reports, and reach the score only through the compliance component. This is why control plane checks in kube-system, which the module keeps producing, do not move the misconfig component.

Excluding a namespace from the score does not exclude it from scanning. Configuration scanners run cluster-wide, and the vulnerability scanner covers every namespace labeled security-scanning.deckhouse.io/enabled, including a system one. What the exclusion controls is only which namespaces the score is calculated over.

Components based on findings

The cve, misconfig and secrets components convert finding counts into a severity load, and the load into a sub-score by the formula 1 / (1 + load / k). The sub-score approaches 0 as findings accumulate but never reaches it, so no single component can zero out the whole score.

The load and the constant k differ per component:

Findings of low and unknown severity do not affect the score. Exposed secrets are scored the most aggressively: a single critical finding already halves the secrets component.

Pod security component

The psa component combines the strictness of the policy applied to a namespace with the way admission-policy-engine enforces it. Strictness comes from the security.deckhouse.io/pod-policy label, or from the podSecurityStandards.defaultPolicy parameter when the label is not set:

• Restricted — 1;
• Baseline — 0.6;
• Privileged — 0.

The result is multiplied by the coefficient of the enforcement mode:

• Deny — 1;
• Warn — 0.5;
• Dryrun — 0.2;
• admission-policy-engine disabled — 0.1.

A Restricted policy in Dryrun mode therefore scores far lower than the same policy in Deny mode. The final and the pre-multiplier values are published in the value and raw fields of the component itself, while the enforcement mode and its coefficient are published once, in the modules.admissionPolicyEngine block.

Namespace score

The namespace score uses the same components, weights and constants as the cluster score, except for compliance: compliance reports are cluster-scoped, so their result cannot be attributed to a namespace. Node vulnerabilities are excluded for the same reason. Because the constants are shared, a namespace that holds all of the findings in the cluster ends up with the same component values as the cluster itself.

The compliance component means the cluster score and the namespace average in the average field are calculated over different sets of components. Never compare those two values with each other, even when every shared component matches.

The weighted average is renormalized over the weights actually used, so a namespace whose every applicable component equals 1 also scores 100.

A component based on reports carries the notMeasured field when no report of the corresponding type exists. The field is not called unknown because the finding counters next to it already use unknown for the number of findings of unknown severity. The rule is the same for the cluster and for a namespace: without reports the component is excluded from the calculation instead of counting as a clean result. A cluster where nothing is scanned therefore no longer collects the full cve and secrets weight for an absence of findings.

The compliance component is a deliberate exception to that rule: with no compliance report deployed it scores 1 and keeps its weight. Compliance reports are enabled on demand through the complianceReports.enabled parameter, so their absence is a configuration choice rather than an unobserved area. This is one more reason not to compare the cluster score with the namespace average.

A namespace with no reports at all is limited to 60: coverage contributes nothing, and only the psa and network weights remain.

This is not a cap on unscanned namespaces. ConfigAuditReport resources are created regardless of the security-scanning.deckhouse.io/enabled label, so an unscanned namespace usually has a known misconfig component and a larger set of applicable weights. What such a namespace always loses is the entire coverage weight, which alone puts a score of 100 out of reach.

The same rule produces the opposite case. A namespace that is labeled for scanning, carries a NetworkPolicy and applies the Restricted policy, but has no report yet, scores 100: the three report-backed components are excluded from the calculation, and the remaining ones are all at their maximum. Right after scanning is enabled, and until the first reports arrive, such a namespace therefore reads as perfect.

Read the score together with the notMeasured field of the components: a value of 100 with cve, misconfig or secrets marked as not measured means the namespace is configured well and nothing has been checked yet, not that the namespace is clean.

Namespace scores live in resources of their own, so their number is not capped. The score.namespaceScores block of ClusterSecurityScore holds a summary only: total is the number of scored namespaces and average is their average score. For interfaces that are not allowed to list SecurityScore across the cluster, the same resource keeps the score.topNamespaces block with the namespaces that need attention first.

A SecurityScore resource is deleted together with its namespace. When a namespace stops being scored — for example, after it gets the security.deckhouse.io/skip-compliance label — the module deletes the resource itself.

Viewing the score

To read the current data, use the following command:

# Scores of every namespace you have access to.
d8 k get securityscore --all-namespaces

# Score of a single namespace with all its components.
d8 k -n <NAMESPACE> get securityscore security-score -o yaml

# Cluster score.
d8 k get clustersecurityscore cluster -o yaml

The output of the first command lists the score, the scanning state and the applied pod security standard for every namespace. The full resources carry the value of every component and the findings it was calculated from.

The resources have the short names secscore and clustersecscore. Use either those or the full name with the group (securityscores.deckhouse.io): a short name without the group may match a resource of another module.

Score metrics

The module exports the cluster score to Prometheus, so the value can be graphed and alerted on without reading the resources.

The following metrics are available:

• deckhouse_trivy_security_score — cluster score from 0 to 100;
• deckhouse_trivy_security_score_component — value of a single component from 0 to 1, with the component name in the component label;
• deckhouse_trivy_security_namespace_score_average — average score across all user namespaces;
• deckhouse_trivy_security_namespaces_total — number of user namespaces;
• deckhouse_trivy_security_namespaces_scanned — number of user namespaces with scanning enabled.

A component that was not measured is not exported at all: the series for it disappears instead of reporting a value. Build an alert on deckhouse_trivy_security_score_component so that it covers that case — pair the condition with absent(), or rely on coverage, which is always present. An expression such as deckhouse_trivy_security_score_component{component="cve"} < 0.5 never fires on a cluster where nothing is scanned, which is where the alert matters most.

Per-namespace scores are not exported as metrics: on a large cluster that would add a time series for every namespace. Read them from the SecurityScore resources instead.

Formula version

Weights, constants and calculation rules stay stable across releases. Any change to them raises the formulaVersion field published next to the score, both for the cluster and for namespaces. A rule change counts as well as a weight change: version 3, for example, differs from version 2 only in that a component without reports is excluded from the average instead of counting as a clean result.

Scores calculated with different formula versions are not comparable.

Working with private and insecure registries

By default the module expects all container registries to serve trusted TLS certificates. For environments that use self-signed certificates, plain HTTP, or that disable TLS verification, the following parameters are available.

Custom CA certificates (additionalRegistryCA)

Use additionalRegistryCA to specify root certificates for private registries.

Each entry has a name (human-readable label, purely informational) and ca (PEM-encoded certificate or chain). When specifying a chain, concatenate the certificates in PEM format with no extra blank lines between them.

spec:
  settings:
    additionalRegistryCA:
      - name: corporate CA
        ca: |
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----
      - name: CA with intermediate
        ca: |
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----
          -----BEGIN CERTIFICATE-----
          ...
          -----END CERTIFICATE-----

Insecure registries (insecureRegistries)

Use insecureRegistries to allow the scanner to connect to specific registries over plain HTTP or HTTPS with unverified certificates. Each entry is a registry hostname (optionally with port).

spec:
  settings:
    insecureRegistries:
      - my.internal-registry.example.com
      - legacy-registry.example.com:5000

This setting applies to workload scanning (scan jobs), admission-time checking (denyVulnerableImages), and registry scanning (RegistryScanTarget).

Database management

The module automatically downloads, maintains, and caches the required databases during scans. Data is kept current through periodic automatic updates.

In air-gapped environments you can mirror database images and use them offline with the d8 CLI utility (via the d8 mirror command).

The vulnerabilityDatabaseMaxAge parameter (default 168h) sets how old trivy-db may be before the TrivyVulnerabilityDatabaseOutdated alert fires. The related TrivyVulnerabilityDatabaseMissing alert fires when the trivy-db-info ConfigMap is absent or has no trivy-db.updatedAt stamp for 30 minutes. Set the parameter to "" to disable both database freshness alerts.