The module lifecycle stage: Generally available version
The module has requirements for installation
v1.3.3
Features
- Placement of the built-in Loki. The new
loki.nodeSelectorandloki.tolerationsmodule parameters place the built-in log storage on dedicated nodes, as the standalonelokimodule allows; they apply to all Loki pods in both modes. Without them Loki is placed automatically, as before. When moving from thelokimodule, copy its placement settings into these parameters, as the migration guide now describes, so Loki stays on the same nodes.
Enhancements
- The PostgreSQL dashboard and the
PgReplicationStatusalert now show replica names instead of only IP addresses, so a standby in another cluster can be told apart from the local replicas. The replication lag phases are shown per replica instead of as a sum over all replicas.
Bug fixes
- Fixed alertmanager in high availability mode not sending notifications, and a notification lost when the first replica failed to deliver it. A restarted or updated alertmanager replica also no longer sends again the notifications that were already delivered.
- Fixed false
NodeDiskBytesUsageandNodeDiskInodesUsagealerts on nodes where containerd keeps container image layers as read-only erofs mounts: the agent no longer collects disk metrics for these mounts. - Fixes in the experimental alert diagnostics: metric checks of reports in a namespace work again, and a report no longer fails on a module without ModuleConfig, no longer names a healthy dependency as failing and no longer describes a registry rate limit as an unreachable registry. The
D8UpmeterProbeGarbagePodsFromDeploymentsalert now gets a diagnostic report. - The
ObservabilityMonitoringPingModuleEnabledalert now tells to check the external ping targets first and then disable themonitoring-pingmodule withd8 platform module disable monitoring-ping, which works also when the module has no ModuleConfig.
v1.3.2
Changes
- In a cluster where the Deckhouse module gateway is enabled, external access to metrics (
externalMetricsAccess) is now published through the Gateway API resourcesListenerSetandHTTPRouteinstead of an Ingress. Clusters that publish modules through an Ingress are not affected.
Bug fixes
- Fixes in the experimental alert diagnostics.
v1.3.1
Bug fixes
- Fixed alerts of a namespace getting stuck in the module forever when they arrived while that namespace was being registered in alertmanager. Prometheus then failed to deliver alerts of the whole batch, notifications about them did not arrive, and
kubectl get observabilityalertsshowed one alert as two objects flipping between Firing and Resolved. A namespace that is stuck or whose notification configuration cannot be loaded no longer holds back the alerts of the other namespaces, and a restart of an alertmanager replica no longer duplicates an alert or recreates its object.
v1.3.0
Features
- Alert diagnostics (experimental). A firing alert can now be answered with a diagnostic report instead of a manual investigation: the module finds the object the alert is about, collects its state — events, logs, owner, probes, resource use, the node it runs on — and, when a diagnostic playbook recognizes the picture, names the cause with the facts behind it and the actions to take. Where the cause cannot be established, the report says so instead of guessing; the result is deterministic, no language model is involved. The report is a Kubernetes resource:
ObservabilityDiagnosticReportin the namespace of the alert orClusterObservabilityDiagnosticReportfor a cluster alert, and it can be rerun. The module ships diagnostics for 182 alerts, each verified on a live cluster, and you can add your own playbooks asClusterObservabilityDiagnosticPlaybookresources. A report shows only the data of the namespace it belongs to, so a user sees nothing they may not read otherwise. The diagnostics is off by default and is turned on withalertDiagnostics.enabled; two parts of it can be turned off separately — running the fixed commands of playbooks in containers (alertDiagnostics.podExec.enabled) and the node agent that reads node state such as the journal, systemd units and disk usage (alertDiagnostics.nodeAgent.enabled). See the “Alert diagnostics” documentation page.
Changes
- The
PgMaxConnectionsandPgWalArchiverFailsalerts now carry the label of the PostgreSQL pod they are about. An alert of these two that is firing while the module is updated resolves and fires again once with the new set of labels, so a repeated notification about it arrives. - Alerts, rules and targets of a namespace are now shown right after access to that namespace is granted. Previously the namespace stayed invisible in these lists until the cached access scope of the token was refreshed.
v1.2.3
Features
- Reminder to disable the
monitoring-pingmodule. Node ping has been collected by the observability agent since v1.2.2, and the agent does not ping on nodes where amonitoring-pingpod still runs. The newObservabilityMonitoringPingModuleEnabledalert asks to disable that module and tells how to check that the external ping targets were carried over first, so the switch-over is not left half done.
Changes
- The access control roles of the module follow the Deckhouse role model: on DKP 1.78 and later they are named
d8:system-capability:observability:edit|viewfor cluster-level resources andd8:namespace-capability:observability:edit|viewfor namespace-level ones, while earlier versions keep the previous names. Check the names used in your bindings against the “Permissions” documentation page.
Bug fixes
- Log storage alerts no longer keep the administrator waiting. A stalled migration of the log volume, a volume left behind by a previous setup and a disk that has stopped being reclaimed are now reported within minutes of the problem appearing instead of half an hour to an hour later, and the disk alert no longer names a waiting time that did not match when it actually fired.
v1.2.2
Features
- Ping monitoring of cluster nodes and external hosts. The opagent agent now measures ICMP packet loss from every node to every other cluster node and raises the
NodePingPacketLossalert when the loss to a node exceeds 5%. External hosts can be added with the newopagent.pingExternalTargetsmodule parameter. The module takes this function over from themonitoring-pingmodule: whilemonitoring-pingis enabled, its checks and its alert keep working and the agent stays silent; a non-emptyexternalTargetslist ofmonitoring-pingis copied into the new parameter automatically once, and if the list does not fit the parameter limits, theObservabilityPingExternalTargetsNotMigratedalert asks to set the parameter by hand.
Bug fixes
- Fixed a module hook that could stall the whole Deckhouse task queue when a dashboard was deleted during another module’s upgrade: the module stayed in
WaitForSynchronizationuntil the Deckhouse leader pod was restarted. - Fixed a memory leak in the opagent agent: the metrics state of recreated pods was never released, so the agent’s memory grew with every pod recreation on the node.
- Fixed the opagent agent still not starting on nodes with the CRI-O container runtime: the v1.2.1 fix added the mount point under one spelling of the CRI-O directory, while the container runtime required the other.
v1.2.1
Features
- Monitoring of managed OpenSearch, Kafka and Valkey. The module now collects metrics of OpenSearch, Kafka and Valkey instances of the corresponding managed service modules. Their dashboards are available in “Monitoring” → “Explore data”, and OpenSearch also gets alerts on metric collection failures, split brain and unassigned shards. Requirements for each service are listed on the “Service monitoring” documentation page.
Enhancements
- The opagent agent uses less memory.
Bug fixes
- Fixed the opagent agent not starting on nodes with the CRI-O container runtime (since v1.1.12).
v1.2.0
Features
- Built-in log storage. The module can now run its own Loki-based log storage, enabled with the
loki.enabledparameter. It works in theSinglemode (one instance, data on a disk or in S3) or theDistributedmode (scalable components, data in S3), and comes with alerts and dashboards. Logs already collected by the deprecatedlokimodule can be migrated together with its volume and remain available for queries; once the built-in storage is enabled, Grafana, log-based metrics and namespace-scoped log access use it. See the “Log storage” documentation page.
Bug fixes
- Restarting the module controller no longer rewrites the status of all metrics rules groups and their PrometheusRule resources, which reduces load on the Kubernetes API server. The
status.lastSyncTimefield now shows the time of the last sync that changed something. - Fixed and clarified the descriptions of the module custom resources.
v1.1.14
Security
- Fixed CVE-2026-116585, CVE-2026-116586, CVE-2026-116587, CVE-2026-78661 and CVE-2026-97045 in the module images and hooks.
v1.1.13
Security
- The health port of the opagent agent (TCP/4293) is no longer reachable from the node network: the agent listens on loopback only, where the kubelet probes reach it. Module settings do not change.
v1.1.12
Features
- Node metrics from the module agent. The opagent agent now collects node metrics itself and replaces node-exporter from the
monitoring-kubernetesmodule: node-exporter is removed once the agent is ready on every node, and node metrics are not duplicated during the switch. The agent now runs on every node of the cluster, so thedop.nodeAffinityparameter is removed from the module settings automatically. External clients can now also write cluster-wide metrics, which are visible only to users with access to system metrics. - Added the
highAvailabilitymodule parameter for manual control of the high availability mode. By default, the mode now follows the globalhighAvailabilityparameter or, if it is not set, the high availability of the cluster control plane. Thealert-kube-apiandobservability-webhookcomponents now always run in two replicas.
Changes
- Zabbix notification channels now take TLS certificates and keys as PEM text in the
ca,certandkeyfields. ThecaFile,certFileandkeyFilefields are now ignored: channels that use them must be updated. A Zabbix certificate without a Subject Alternative Name can be accepted withskipServerNameVerification. Changes to a firing alert group are now sent within 10 seconds.
Bug fixes
- Fixed the instructions for manual import of the Zabbix template: the file must be imported twice, first as a template and then as a host.
v1.1.11
Security
- Fixed CVE-2026-85747 in the module hooks.
- Fixed CVE-2026-111943 and CVE-2026-63209 in the module images.
v1.1.10
Security
- Fixed CVE-2026-56851, CVE-2026-56857, CVE-2026-56866, CVE-2026-78659, CVE-2026-78660, CVE-2026-78663, CVE-2026-78667, CVE-2026-78669, CVE-2026-94439, CVE-2026-94440, CVE-2026-94448, CVE-2026-97030, CVE-2026-97031 and CVE-2026-97032 in the module images.
v1.1.9
Changes
- The alert-kube-api component now logs at the standard verbosity level instead of the debug level, which reduces its log volume.
Bug fixes
- Fixed a memory and goroutine leak in the lightweight logs service: every log query opened a new connection to the reader on each node and never closed it, so the memory usage of the service grew with the number of queries. Connections are now reused between queries. A label query (for example, autocomplete in Grafana) also no longer fails entirely when the reader on a single node is unavailable: the result is built from the remaining nodes.
- The dashboard storage service no longer crashes on a dashboard resource without content.
- Grafana: a request that hit an internal error is now answered with a single error response instead of a corrupted one; queries to Azure Monitor, Azure Log Analytics and InfluxDB data sources with missing or unexpected fields now return an error or text values instead of failing the Grafana backend.
v1.1.8
Security
- Grafana vulnerabilities CVE-2026-42129, CVE-2026-10601 (path traversal in the Loki data source) and CVE-2026-42127 (denial of service in public dashboards), fixed in the Grafana image since v1.1.7, are now declared to vulnerability scanners. CVE-2026-9029 does not affect the module.
v1.1.7
Features
- Zabbix notification channel. Alerts can now be delivered directly to a Zabbix Server or Zabbix Proxy: a notification channel of type
Zabbixpushes them over the Zabbix sender protocol, encrypted by default. The channel comes with a ready-to-import Zabbix template that already contains the cluster host and host group, so no items or triggers need to be created by hand. The template is placed in a ConfigMap next to the channel and, if the Zabbix API address and token are specified in the channel, is imported into Zabbix automatically (Zabbix 6.4 or newer). Resolved alerts clear their Zabbix triggers immediately, and theDeadMansSwitchalert keeps a heartbeat item up to date so that Zabbix detects a broken delivery path.
Bug fixes
- Moving a dashboard to another folder no longer shows a spurious
editablechange on save, and dashboard changes made after leaving edit mode are no longer lost from the save diff and the unsaved-changes warning. The dashboard JSON model can now be edited in the dashboard settings.
v1.1.6
Changes
- The metrics collection agent (opagent) now runs only on nodes that host monitored services instead of every node in the cluster.
v1.1.5
Changes
- The CA certificate of the module admission webhook is now stored in a dedicated Secret and is no longer regenerated after a Deckhouse restart. Previously a restart could leave the webhook configuration with a CA that did not match the certificate served by the webhook, and creating or updating module resources failed until the certificate was reissued.
Bug fixes
- Metrics without a
namespacelabel (node-exporter, kubelet and other node-level metrics) are no longer hidden from the labels API for users with cluster-wide metrics access. Grafana metric autocomplete and dashboard variables such asmetrics(node_.*)andlabel_values(nodename)now list them consistently with query results. - Users can now create ClusterObservabilityPropagatedMetricsRulesGroup resources — shared metrics rules groups propagated to all projects. Previously their creation was rejected.
- Fixed Grafana lacking permission to read the sensitive fields of GrafanaAdditionalDatasource resources.
- External links in dashboard text panels now open in a new browser tab instead of replacing the embedded Grafana view.
Security
- Fixed various vulnerabilities.
v1.1.4
Changes
- System metrics of the module components now carry the
d8_source="dkp"label. This prevents duplicate time series and failing rule evaluations when a user exporter exposes metrics with the same names. - The module namespace is now checked against the
restrictedPod Security Standards profile. Violations are reported as warnings only and do not block workloads. The module now requires Deckhouse 1.74 or newer.
Enhancements
- Added PostgreSQL connection setup instructions to the service monitoring documentation, including how to reserve connections for the monitoring role on PostgreSQL 16 and newer.
Bug fixes
- Removed the default value of the evaluation interval for propagated metrics rules groups.
- Added RBAC v1 permissions for users on ObservabilityLogsRulesGroup and ClusterObservabilityLogsRulesGroup resources.
- Fixed an error that caused the metrics collection agent to repeatedly attempt to update its own components in the cluster.
Security
- Notification channel credentials are now masked in API responses. Reading them requires a dedicated permission, so users who may only view a channel no longer see its credentials.
- Notification channel addresses are now validated: a channel can only point at an address that is a legitimate notification target. Addresses inside the cluster and in the private network remain allowed.
- In the
httpConfigsection of notification channels, the fields that reference a file inside the container are deprecated and no longer applied:basicAuth.passwordFile,authorization.credentialsFile,oauth2.clientSecretFile,caFile,certFileandkeyFilein bothtlsConfigsections, andfilesinhttpHeaders. Usevaluesandsecretsto pass header contents. ThefollowRedirectsfield is no longer applied either — redirects are never followed. Existing resources remain valid, but the values of these fields are ignored, so a channel that relied on them needs to be reconfigured. - Hardened validation of module configuration parameters.
- Fixed various vulnerabilities.
v1.1.0
Features
- Added high-availability mode support to observability-controller
- Added Logs ruler service allowing to build prometheus metrics based on logs
- Added alert status field selector to alert-kube-api.
- Added scrape interval configuration to opagent
Changes
- Added source=“deckhouse” label to all system metrics
- Added information about rolebinding and CAR using to module documentation
- Opagent configuration changed to monitor only managed services by default.
Bug fixes
- Fixed broken links in module documentation
- Fixed a bug that caused delays in the appearance of new namespaces in the monitoring Explore data and Dashboards
- Fixed a bug in module documentation specified that lightweight logs service is disabled by default
- Fixed dashboards to properly show metrics if opagent restarts.
- Fixed a bug causing alertmanager to fail if publicDomainTemplate is blank.
- Fixed a bug causing opagent to stop collecting metrics on containerd restart.
v1.0.8
Bug fixes
- Fixed a bug that caused cluster silencers to not work properly after alertmanager restart.
v1.0.7
Changes
- PrometheusUnavailable alert changed severity to S3 and for clause for alert changed to 5 minutes.
v1.0.6
Security
- Fixed various vulnerabilities in third-party libraries.
Bug fixes
- Fixed components placement on a master-only k8s clusters
v1.0.5
Bug fixes
- Module resources deploy order is now deterministic and works properly with nelm
v1.0.4
Changes
- The observability module is now fully compatible with the Deckhouse CSE
v1.0.3
Changes
- Opagent collects metrics for managed services only by default.
Bug fixes
- Fixed an issue that caused opagent to stop collecting metrics after containerd restart.
v1.0.2
Security
- Fixed vulnerability related to CVE-2026-40181 in react-router 6.30.2
v1.0.1
Changes
- Module images are now built using the container-base images and the
pmpackage manager
v1.0.0
Changes
- Changes Observability module stage from Preview to General Availability
Enhancements
- Improved Russian CRD documentation and removed obsolete CRDs
- Added TCP/TLS in Alertmanager Gossip sync protocol.
Features
- Added lightweight logs in module Observability. It allows to select lightweight-logs datasource in Explore Data to explore all namespace pods logs in one place. It supports filtering logs by labels and annotations.
- Managed services monitoring support: it adds managed PostgreSQL or Memcache services monitoring dashboards to Explore Data section of a project.
- Added additional labels support for DKP alerts. This way project alerts may have additional custom labels allowing users to filter them in external system using those labels.
- Added eXpress messenger support in Alertmanager. Now it may be used when setting up notifications channel.
- Added auth config to notifications webhook channel
- Added ability to enable/disable Polk annotations in alerts notifications
Bug fixes
- Fixed write path for external metrics access in module documentation
- Fixed an issue that sometimes allowed upmeter probes test alerts to pass to general alerts list.
- Drop finalizers from the Deckhouse-deployed resources on module disable
- Remove the default value for the interval field in the metrics rule groups specs.
- Enables watch config for opagent daemonset to properly reload SA token
- Alertmanager Telegram and Slack notifier messages splitting for oversized alert groups
v0.9.8
Bug fixes
- Fixed templates to check if required modules are enabled instead of checking if their CRDs
v0.9.7
Bug fixes
- Fixed access to the prometheus remote write endpoint
v0.9.6
Bug fixes
- Fix CVEs related to CVE-2026-42151, CVE-2026-42154, CVE-2026-44903, CVE-2025-15558, CVE-2025-47914, CVE-2025-58181, CVE-2026-32952, stdlib (Go 1.25.10) and CVEs in bundled base images. Additional CVEs in
github.com/prometheus/prometheus,github.com/grafana/tempo,github.com/getkin/kin-openapi,github.com/grafana/grafana-plugin-sdk-goandgithub.com/grafana/grafanaare addressed via VEX statements as not exploitable.
v0.9.5
Bug fixes
- Removed the default value for the interval field in the metrics rule groups specs.
v0.9.4
Bug fixes
- Enabled watch config for opagent daemonset to properly reload SA token. That issue caused D8KubernetesStaleTokensDetected alerts for opagent.
v0.9.3
Bug fixes
- Fixed Prometheus discovery with global HA mode disabled.
v0.9.2
Bug fixes
- Fixed console dashboard panel links.
v0.9.1
Bug fixes
- Fixed vulnerabilities related to CVEs
v0.9.0
Features
- Added logs multitenancy support. Now explore data only shows current project / cluster logs. It allows to setup users access to specific namespaces logs only.
Changes
- High availability mode improved to keep firing and resolved alerts while module services rollouts and updates
- Replaced dashboard uniq UID validation with an alert that notifies about UID conflicts
- Improved module services authorization.
- Observability module documentation updated
Bug fixes
- Fixed naming convention validation for label names, annotation keys, and metrics names in observability rules to reject incompatible resources at apply time
- Fixed an issue caused projects monitoring status alerts and rules not displaying properly
- Fixed issues causing 401 responses affected dashboards with lots of panels and remote write metrics requests.
v0.8.2
Bug Fixes
- Fixed a bug in the Email notification channel where the “Require TLS” setting was ignored.
v0.8.1
Bug fixes
- Fixed vulnerabilities related to CVEs (CVE-2026-33186, CVE-2026-24051, CVE-2025-15558, CVE-2026-25679, CVE-2025-68470)
v0.8.0
Features
- Alerting high availability — when Deckhouse Kubernetes Platform runs in high availability mode it also runs alerting in HA mode for better availability.
- Improved Prometheus availability monitoring — DeadMansSwitch and PrometheusUnavailable alerts were updated for more reliable monitoring of Prometheus availability.
- Propagated notification policies — You can configure alert notification delivery per project using propagated notification policies together with propagated notification channels.
Bug Fixes
- Fixed empty rule group status in metrics rule groups caused by resource version conflicts.
- Fixed module enablement failing when the global
publicDomainTemplatesetting was left blank. - Fixed PromQL queries using negative regexp matcher (
!~) incorrectly returning “No data”. - Fixed incorrect graph rendering in Grafana when multiple conditions were applied to the same label in a PromQL query.
- Fixed errors when using a rule group custom resource with no rules defined.
- Cluster rule expressions now support an empty namespace for metrics selection.
Enhancements
- Deckhouse modules’ Prometheus rules are now stored in the
d8-observabilitynamespace. - Alerts from Prometheus rules that do not specify cluster/namespace scope are now listed under the corresponding namespace’s alerts instead of system alerts. This affects only rules not defined via ClusterObservabilityMetricsRulesGroup or ObservabilityMetricsRulesGroup.
- FAQ section on configuring external Grafana access to metrics was clarified.
Release v0.7.23
Bug fixes
- Fixed an issue caused some Grafana datasource plugins (ClickHouse) to fail to connect to databases.
- User roles permissions fixes
Release v0.7.22
Bug fixes
- Bump Go version to 1.25.6 to fix known vulnerabilities in stdlib
Release v0.7.21
Changes
- Observability alerts API service turned on for Kubernetes v1.32+
Release v0.7.20
Changes
- Expanded the Prometheus rules API
Release v0.7.19
Changes
- Observability webhook was moved to the control plane nodes. The component is now treated as critical for the cluster stability.
Release v0.7.18
Changes
- Removed internal module labels from prometheus recording rules.
Release v0.7.17
Changes
- Temporarily removed namespaces enforcing for cluster rules
Release v0.7.16
Bug fixes
- Fixed an issue which caused namespace deletion stuck.
Release v0.7.15
Changes
- Apiservice v1alpha1.alerts.observability.deckhouse.io turned off for Kubernetes v 1.32
Release v0.7.14
Bug fixes
- Fixed an issue which caused tenant alertmanager configuration load to fail.
Changes
- It now creates PrometheusRules in d8-observability namespace. Creating prometheus rules CRs in d8-monitoring namespace caused it to trigger D8CustomPrometheusRuleFoundInCluster alert.
v0.6.0
Features
- Added dark theme support for the dashboard visualization components. While the main UI has previously supported dark mode, this update extends theme awareness to full support in dashboards and graphs.
- Introduced a new kiosk mode for dashboards. When enabled, all navigation elements and menus are hidden, allowing dashboards to be displayed in full-screen mode — ideal for use on wall-mounted displays or TV screens.
Security
- Fixed vulnerabilities related to CVEs (CVE-2025-30153, CVE-2025-47912, CVE-2025-58183, CVE-2025-58185, CVE-2025-58186, CVE-2025-58187, CVE-2025-58188, CVE-2025-58189, CVE-2025-61723, CVE-2025-61724, CVE-2025-61725)
v0.5.6
Changes
- Changed module statuses from experimental to preview.
Bug Fixes
- Fixed an issue related to authorization token caching that caused incorrect triggering of the built-in alert: D8KubernetesStaleTokensDetected.
v0.5.5
Security
- Fixed vulnerabilities related to CVEs
v0.5.4
Changes
- Prepared for container v2 — added mount points to images as part of the transition groundwork, no impact on current usage.
v0.5.3
Bug Fixes
- Fixed incorrect link mutations in dashboards — previously, some dashboard links could be displayed incorrectly due to a mutation issue.
v0.5.2
Bug Fixes
- Fixed an issue related to authorization token caching that caused incorrect triggering of the built-in alert: D8KubernetesStaleTokensDetected.
v0.5.1
Bug Fixes
- Fixed an issue preventing correct exit from dashboard edit mode.
Features
- Added ability to remote write metrics to cluster Prometheus.