The module lifecycle stage: Generally available version
The module has requirements for installation
The observability module lets you collect metrics and track the state of managed services running in the DKP cluster.
The service monitoring is enabled by default if the following requirements are met:
- DKP version 1.76.0 or higher
promppmodule version 3.7.10 or higher (if this module is used)
Supported services
The module currently supports monitoring of the following managed services:
| Service type | Scope | Who has access |
|---|---|---|
| PostgreSQL | Project (namespace) level | Users of the corresponding project |
| Memcached | Project (namespace) level | Users of the corresponding project |
Connecting a PostgreSQL instance
To let the module collect metrics from a PostgreSQL instance, run the following as a superuser (or another role that already has full access to pg_stat_* views) on that instance:
CREATE ROLE okagent WITH LOGIN;
CREATE SCHEMA okmeter; -- Keeps the helper function separate from the rest of the schema.
GRANT USAGE ON SCHEMA okmeter TO okagent; -- Grants okagent access to the helper schema.
CREATE OR REPLACE FUNCTION okmeter.pg_stats(text) RETURNS SETOF RECORD AS
$$
DECLARE r record;
BEGIN
FOR r IN EXECUTE 'SELECT r FROM pg_' || $1 || ' r' LOOP RETURN NEXT r;
END LOOP;
RETURN;
END
$$ LANGUAGE plpgsql SECURITY DEFINER; -- Lets okagent read restricted pg_stat_* views without extra privileges.The module’s monitoring agent (opagent) reaches PostgreSQL through a local Unix domain socket on the same host, expected at /controller/run, and connects without a password — pg_hba.conf needs to allow okagent to connect locally without a password check.
A simpler alternative on PostgreSQL 10 and newer
Starting with PostgreSQL 10, the built-in pg_monitor role gives the same monitoring visibility as the okmeter.pg_stats function above without creating a schema or a function:
CREATE ROLE okagent WITH LOGIN;
GRANT pg_monitor TO okagent;Use this instead of the schema/function setup if all your instances run PostgreSQL 10 or newer. The okmeter.pg_stats function above additionally works on PostgreSQL versions earlier than 10, where the pg_monitor role doesn’t exist yet.
PostgreSQL 16 and newer
Optional: without a reserved connection, okagent gets rejected like any other client once the instance runs out of connections. Starting with PostgreSQL 16, grant okagent the built-in pg_use_reserved_connections role and reserve at least one connection for it — superuser_reserved_connections alone doesn’t cover non-superuser roles:
GRANT pg_use_reserved_connections TO okagent;
ALTER SYSTEM SET reserved_connections = 1;reserved_connections can only be set at server start, so the PostgreSQL server needs to be restarted for the change to take effect. Make sure reserved_connections plus superuser_reserved_connections stays below max_connections.
One reserved connection is enough — okagent never holds more than one connection to an instance at a time. Not available on PostgreSQL versions earlier than 16.
Dashboards
To view a service’s health data, open the corresponding dashboard in the Deckhouse web UI. For that, go to “Monitoring” → “Explore data”, then choose the service type (for example, “PostgreSQL”) in the vertical menu and select the required service instance from the drop-down list.

Alerts
The following alerts are available for the PostgreSQL services.
| Service type | Alert name | Description |
|---|---|---|
| PostgreSQL | PgAutovacuumWorkers |
Autovacuum worker limit reached; table bloat and degraded query performance are possible |
| PostgreSQL | PgCheckError |
The monitoring agent failed to collect metrics from the PostgreSQL instance |
| PostgreSQL | PgMaxConnections |
Connection pool usage exceeds 95%; new connections may be refused |
| PostgreSQL | PgPluginConfig |
The PostgreSQL monitoring plugin is not fully configured |
| PostgreSQL | PgReplicationStatus |
The primary server lost its replication connection to the standby; data on the standby may be stale |
| PostgreSQL | PgTxidWraparound |
Approaching transaction ID wraparound; the database will shut down unless VACUUM is run urgently |
| PostgreSQL | PgWaitingConnections |
More than one connection is waiting; lock contention may affect performance |
| PostgreSQL | PgWalArchiverFails |
WAL archiver failures; WAL segments may accumulate and disk space may be exhausted |
Disabling monitoring
Service monitoring is controlled by the observability.deckhouse.io/servicemonitoring label. You can set it on a namespace (for all instances in the project) or on a specific pod (for a single instance).
Allowed values:
| Value | Metrics collection | Alerts |
|---|---|---|
enabled |
Yes | Yes |
no-alerts |
Yes | No |
disabled |
No | No |
If the label is missing or has an unrecognized value, enabled is applied.
Configuration example:
apiVersion: v1
kind: Namespace
metadata:
name: project-a
labels:
observability.deckhouse.io/servicemonitoring: no-alertsA pod label can only tighten the restriction set on the namespace, not loosen it. The effective value for an instance is defined based on the strictest rule: enabled < no-alerts < disabled.
The following table shows how the effective value depends on the namespace and pod labels.
| Namespace label | Pod label | Effective value |
|---|---|---|
disabled |
Any value or not set | disabled |
| Any value or not set | disabled |
disabled |
no-alerts |
Any value except disabled |
no-alerts |
Any value except disabled |
no-alerts |
no-alerts |
enabled or not set |
enabled or not set |
enabled |
Where the agent runs
The agent is deployed as a DaemonSet on every node of the cluster. Besides service metrics, it collects node metrics (CPU, memory, disks, network, and so on — it replaces node-exporter), so it has to run even on nodes without monitored services.
As node-exporter did, the agent reads Prometheus text files (*.prom) from the /var/run/node-exporter-textfile directory on the node and exposes them as node metrics with the job="node-exporter" label. Anything that wrote custom node metrics into this directory keeps working without changes.
While the node-exporter of the monitoring-kubernetes module is still running on a node, the agent on that node does not export node metrics: the two never collect the same series at once. monitoring-kubernetes 1.1.0 and newer removes its node-exporter once the agent has been rolled out on every node, and the agent on each node takes over at its next collection. An older monitoring-kubernetes keeps its node-exporter, node metrics continue to come from it, and the agent collects service metrics only. No particular version of monitoring-kubernetes is required.
Node metrics
Node metrics are provided by the node_exporter plugin embedded in the agent. Its series carry job="node-exporter" and the labels tier="cluster", d8_source="dkp" and node=<node name>, and have the same names and labels as the node-exporter DaemonSet of the monitoring-kubernetes module. Existing DKP dashboards and alerts built on node_* metrics keep working without changes.
The agent’s own legacy host metrics are not collected in the module: cpu__*, memory__*, disk__*, net__*, netstat__protocol__packets__*, load_average__* and the unprefixed CPU, memory and network counters (user, idle, total, free, in_bytes, out_bytes, and so on). If you have your own queries on these metrics, rewrite them to node_exporter expressions using the table below. Two things change on the way:
- Units. The agent published percentages (
cpu__usage__*) and ready per-second rates (disk__device__*,net__interface__*,swap__io__*); node_exporter publishes raw counters, so the expression needsrate()over a window —[R]in the table, for example$__rate_intervalin dashboards or5min alert rules. - Labels.
mount_pointbecomesmountpoint, the block devicedevice="/dev/sda"becomesdevice="sda"(there is nopci_bdf), the networkinterfacebecomesdevice. If a legend, threshold orby()grouping depends on the old label name, rename it on the node_exporter side withlabel_replace, for examplelabel_replace(<expr>, "mount_point", "$1", "mountpoint", "(.*)").
All node_exporter selectors below also need job="node-exporter"; it is omitted for brevity. To keep the history recorded before the upgrade in the same panel, combine both sides: (<node_exporter expression>) or on(instance) (<legacy expression>) — a host that has node_* series uses them, a host that has only legacy series falls back to them.
| Family | Legacy metric | Use instead |
|---|---|---|
| CPU | cpu__usage__<mode> (user, system, nice, iowait, irq, softirq, steal), % |
100 * sum by(instance)(rate(node_cpu_seconds_total{mode="<mode>"}[R])) / sum by(instance)(rate(node_cpu_seconds_total[R])) |
| CPU | user, system, nice, idle, iowait, irq, softirq, steal (job="cpu"), seconds |
sum by(instance)(node_cpu_seconds_total{mode="<mode>"}) |
| CPU | cpu__logical_cores |
count by(instance)(node_cpu_seconds_total{mode="idle"}) |
| CPU | cpu__frequency__mhz{core} |
node_cpu_scaling_frequency_hertz / 1e6 (label cpu instead of core) |
| CPU | cpu__governor{core, governor} |
node_cpu_scaling_governor (label cpu instead of core) |
| CPU | cpu__thermal_throttle__core_throttle_count, cpu__thermal_throttle__package_throttle_count |
node_cpu_core_throttles_total (labels package, core), node_cpu_package_throttles_total |
| Memory and swap | memory__total, total |
node_memory_MemTotal_bytes |
| Memory and swap | memory__available, available |
node_memory_MemAvailable_bytes |
| Memory and swap | memory__usage__free, free |
node_memory_MemFree_bytes |
| Memory and swap | memory__usage__buffers, buffers |
node_memory_Buffers_bytes |
| Memory and swap | memory__usage__cached, cached |
node_memory_Cached_bytes |
| Memory and swap | memory__usage__used |
node_memory_MemTotal_bytes - node_memory_MemFree_bytes - node_memory_Buffers_bytes - node_memory_Cached_bytes |
| Memory and swap | memory__errors{type="correctable"}, memory__errors{type="uncorrectable"} |
sum by(instance)(node_edac_correctable_errors_total), sum by(instance)(node_edac_uncorrectable_errors_total) |
| Memory and swap | swap__total, swap_total |
node_memory_SwapTotal_bytes |
| Memory and swap | swap__space__free, swap_free |
node_memory_SwapFree_bytes |
| Memory and swap | swap__space__used |
node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes |
| Memory and swap | swap_in, swap_out (counters) |
node_vmstat_pswpin * 4096, node_vmstat_pswpout * 4096 |
| Memory and swap | swap__io__in, swap__io__out (per second) |
rate(node_vmstat_pswpin[R]) * 4096, rate(node_vmstat_pswpout[R]) * 4096 |
| File systems | bytes_total |
node_filesystem_size_bytes |
| File systems | bytes_used, disk__partition__bytes_used, disk__partition__space__used |
node_filesystem_size_bytes - node_filesystem_free_bytes |
| File systems | disk__partition__bytes_available, disk__partition__space__available |
node_filesystem_avail_bytes |
| File systems | disk__partition__bytes_reserved, disk__partition__space__reserved |
node_filesystem_free_bytes - node_filesystem_avail_bytes |
| File systems | inodes_total |
node_filesystem_files |
| File systems | inodes_used, disk__partition__inodes__used |
node_filesystem_files - node_filesystem_files_free |
| File systems | disk__partition__inodes__free |
node_filesystem_files_free |
| File systems | disk__partition__read_write |
1 - node_filesystem_readonly |
| Block devices | disk__device__ops__read, disk__device__ops__write |
rate(node_disk_reads_completed_total[R]), rate(node_disk_writes_completed_total[R]) |
| Block devices | disk__device__bytes__read, disk__device__bytes__write |
rate(node_disk_read_bytes_total[R]), rate(node_disk_written_bytes_total[R]) |
| Block devices | disk__device__io_time__read, disk__device__io_time__write |
rate(node_disk_read_time_seconds_total[R]), rate(node_disk_write_time_seconds_total[R]) |
| Block devices | disk__device__io_time__total |
rate(node_disk_io_time_seconds_total[R]) (× 100 = utilization, %) |
| Network interfaces | in_bytes, in_packets, in_errors, in_dropped |
node_network_receive_bytes_total, node_network_receive_packets_total, node_network_receive_errs_total, node_network_receive_drop_total |
| Network interfaces | out_bytes, out_packets, out_errors, out_dropped |
node_network_transmit_bytes_total, node_network_transmit_packets_total, node_network_transmit_errs_total, node_network_transmit_drop_total |
| Network interfaces | net__interface__bytes__in / __out, net__interface__packets__in / __out, net__interface__errors__in / __out, net__interface__drops__in / __out |
rate(<matching counter above>[R]) |
| Network interfaces | mtu |
node_network_mtu_bytes |
| Conntrack | net__conntrack__count |
node_nf_conntrack_entries |
| Conntrack | net__conntrack__max |
node_nf_conntrack_entries_limit |
| Netstat packets | netstat__protocol__packets__in{protocol="tcp"}, {protocol="udp"}, {protocol="icmp"} |
node_netstat_Tcp_InSegs, node_netstat_Udp_InDatagrams, node_netstat_Icmp_InMsgs |
| Netstat packets | netstat__protocol__packets__out{protocol="tcp"}, {protocol="udp"}, {protocol="icmp"} |
node_netstat_Tcp_OutSegs, node_netstat_Udp_OutDatagrams, node_netstat_Icmp_OutMsgs |
| Netstat packets | netstat__protocol__packets__retransmits{protocol="tcp"} |
node_netstat_Tcp_RetransSegs |
| Load average | load_average__1min, ___1min |
node_load1 |
| Load average | load_average__5min, ___5min |
node_load5 |
| Load average | load_average__15min, ___15min |
node_load15 |
The node_exporter netstat metrics have no protocol label; add it with label_replace(<expr>, "protocol", "tcp", "", "") if a query groups by it. Everything else the agent collects on the node — uname, time__*, raid_*, disk_smart_*, netstat__connections__*, netstat__listen__*, netstat__protocol__packets__errors, cpu_flags, cpu__physical_*, memory__numa__allocated, disk__fs__errors, disk__partition__ops__*, disk__partition__bytes__*, the text metrics fs, device, mount_options, hw_address, ipv4_addresses, the interface status status{job="net"}, net__interface__status, net__ip__status, and process metrics — is still collected under the old names.