The module lifecycle stage: Generally available version

The module has requirements for installation

The observability module lets you collect metrics and track the state of managed services running in the DKP cluster.

The service monitoring is enabled by default if the following requirements are met:

  • DKP version 1.76.0 or higher
  • prompp module version 3.7.10 or higher (if this module is used)

Supported services

The module currently supports monitoring of the following managed services:

Service type Scope Who has access
PostgreSQL Project (namespace) level Users of the corresponding project
Memcached Project (namespace) level Users of the corresponding project

Connecting a PostgreSQL instance

To let the module collect metrics from a PostgreSQL instance, run the following as a superuser (or another role that already has full access to pg_stat_* views) on that instance:

CREATE ROLE okagent WITH LOGIN;
CREATE SCHEMA okmeter; -- Keeps the helper function separate from the rest of the schema.
GRANT USAGE ON SCHEMA okmeter TO okagent; -- Grants okagent access to the helper schema.
CREATE OR REPLACE FUNCTION okmeter.pg_stats(text) RETURNS SETOF RECORD AS
$$
DECLARE r record;
BEGIN
    FOR r IN EXECUTE 'SELECT r FROM pg_' || $1 || ' r' LOOP RETURN NEXT r;
    END LOOP;
    RETURN;
END
$$ LANGUAGE plpgsql SECURITY DEFINER; -- Lets okagent read restricted pg_stat_* views without extra privileges.

The module’s monitoring agent (opagent) reaches PostgreSQL through a local Unix domain socket on the same host, expected at /controller/run, and connects without a password — pg_hba.conf needs to allow okagent to connect locally without a password check.

A simpler alternative on PostgreSQL 10 and newer

Starting with PostgreSQL 10, the built-in pg_monitor role gives the same monitoring visibility as the okmeter.pg_stats function above without creating a schema or a function:

CREATE ROLE okagent WITH LOGIN;
GRANT pg_monitor TO okagent;

Use this instead of the schema/function setup if all your instances run PostgreSQL 10 or newer. The okmeter.pg_stats function above additionally works on PostgreSQL versions earlier than 10, where the pg_monitor role doesn’t exist yet.

PostgreSQL 16 and newer

Optional: without a reserved connection, okagent gets rejected like any other client once the instance runs out of connections. Starting with PostgreSQL 16, grant okagent the built-in pg_use_reserved_connections role and reserve at least one connection for it — superuser_reserved_connections alone doesn’t cover non-superuser roles:

GRANT pg_use_reserved_connections TO okagent;
ALTER SYSTEM SET reserved_connections = 1;

reserved_connections can only be set at server start, so the PostgreSQL server needs to be restarted for the change to take effect. Make sure reserved_connections plus superuser_reserved_connections stays below max_connections.

One reserved connection is enough — okagent never holds more than one connection to an instance at a time. Not available on PostgreSQL versions earlier than 16.

Dashboards

To view a service’s health data, open the corresponding dashboard in the Deckhouse web UI. For that, go to “Monitoring” → “Explore data”, then choose the service type (for example, “PostgreSQL”) in the vertical menu and select the required service instance from the drop-down list.

Viewing the service monitoring data

Alerts

The following alerts are available for the PostgreSQL services.

Service type Alert name Description
PostgreSQL PgAutovacuumWorkers Autovacuum worker limit reached; table bloat and degraded query performance are possible
PostgreSQL PgCheckError The monitoring agent failed to collect metrics from the PostgreSQL instance
PostgreSQL PgMaxConnections Connection pool usage exceeds 95%; new connections may be refused
PostgreSQL PgPluginConfig The PostgreSQL monitoring plugin is not fully configured
PostgreSQL PgReplicationStatus The primary server lost its replication connection to the standby; data on the standby may be stale
PostgreSQL PgTxidWraparound Approaching transaction ID wraparound; the database will shut down unless VACUUM is run urgently
PostgreSQL PgWaitingConnections More than one connection is waiting; lock contention may affect performance
PostgreSQL PgWalArchiverFails WAL archiver failures; WAL segments may accumulate and disk space may be exhausted

Disabling monitoring

Service monitoring is controlled by the observability.deckhouse.io/servicemonitoring label. You can set it on a namespace (for all instances in the project) or on a specific pod (for a single instance).

Allowed values:

Value Metrics collection Alerts
enabled Yes Yes
no-alerts Yes No
disabled No No

If the label is missing or has an unrecognized value, enabled is applied.

Configuration example:

apiVersion: v1
kind: Namespace
metadata:
  name: project-a
  labels:
    observability.deckhouse.io/servicemonitoring: no-alerts

A pod label can only tighten the restriction set on the namespace, not loosen it. The effective value for an instance is defined based on the strictest rule: enabled < no-alerts < disabled. The following table shows how the effective value depends on the namespace and pod labels.

Namespace label Pod label Effective value
disabled Any value or not set disabled
Any value or not set disabled disabled
no-alerts Any value except disabled no-alerts
Any value except disabled no-alerts no-alerts
enabled or not set enabled or not set enabled

Where the agent runs

The agent is deployed as a DaemonSet on every node of the cluster. Besides service metrics, it collects node metrics (CPU, memory, disks, network, and so on — it replaces node-exporter), so it has to run even on nodes without monitored services.

As node-exporter did, the agent reads Prometheus text files (*.prom) from the /var/run/node-exporter-textfile directory on the node and exposes them as node metrics with the job="node-exporter" label. Anything that wrote custom node metrics into this directory keeps working without changes.

While the node-exporter of the monitoring-kubernetes module is still running on a node, the agent on that node does not export node metrics: the two never collect the same series at once. monitoring-kubernetes 1.1.0 and newer removes its node-exporter once the agent has been rolled out on every node, and the agent on each node takes over at its next collection. An older monitoring-kubernetes keeps its node-exporter, node metrics continue to come from it, and the agent collects service metrics only. No particular version of monitoring-kubernetes is required.

Node metrics

Node metrics are provided by the node_exporter plugin embedded in the agent. Its series carry job="node-exporter" and the labels tier="cluster", d8_source="dkp" and node=<node name>, and have the same names and labels as the node-exporter DaemonSet of the monitoring-kubernetes module. Existing DKP dashboards and alerts built on node_* metrics keep working without changes.

The agent’s own legacy host metrics are not collected in the module: cpu__*, memory__*, disk__*, net__*, netstat__protocol__packets__*, load_average__* and the unprefixed CPU, memory and network counters (user, idle, total, free, in_bytes, out_bytes, and so on). If you have your own queries on these metrics, rewrite them to node_exporter expressions using the table below. Two things change on the way:

  • Units. The agent published percentages (cpu__usage__*) and ready per-second rates (disk__device__*, net__interface__*, swap__io__*); node_exporter publishes raw counters, so the expression needs rate() over a window — [R] in the table, for example $__rate_interval in dashboards or 5m in alert rules.
  • Labels. mount_point becomes mountpoint, the block device device="/dev/sda" becomes device="sda" (there is no pci_bdf), the network interface becomes device. If a legend, threshold or by() grouping depends on the old label name, rename it on the node_exporter side with label_replace, for example label_replace(<expr>, "mount_point", "$1", "mountpoint", "(.*)").

All node_exporter selectors below also need job="node-exporter"; it is omitted for brevity. To keep the history recorded before the upgrade in the same panel, combine both sides: (<node_exporter expression>) or on(instance) (<legacy expression>) — a host that has node_* series uses them, a host that has only legacy series falls back to them.

Family Legacy metric Use instead
CPU cpu__usage__<mode> (user, system, nice, iowait, irq, softirq, steal), % 100 * sum by(instance)(rate(node_cpu_seconds_total{mode="<mode>"}[R])) / sum by(instance)(rate(node_cpu_seconds_total[R]))
CPU user, system, nice, idle, iowait, irq, softirq, steal (job="cpu"), seconds sum by(instance)(node_cpu_seconds_total{mode="<mode>"})
CPU cpu__logical_cores count by(instance)(node_cpu_seconds_total{mode="idle"})
CPU cpu__frequency__mhz{core} node_cpu_scaling_frequency_hertz / 1e6 (label cpu instead of core)
CPU cpu__governor{core, governor} node_cpu_scaling_governor (label cpu instead of core)
CPU cpu__thermal_throttle__core_throttle_count, cpu__thermal_throttle__package_throttle_count node_cpu_core_throttles_total (labels package, core), node_cpu_package_throttles_total
Memory and swap memory__total, total node_memory_MemTotal_bytes
Memory and swap memory__available, available node_memory_MemAvailable_bytes
Memory and swap memory__usage__free, free node_memory_MemFree_bytes
Memory and swap memory__usage__buffers, buffers node_memory_Buffers_bytes
Memory and swap memory__usage__cached, cached node_memory_Cached_bytes
Memory and swap memory__usage__used node_memory_MemTotal_bytes - node_memory_MemFree_bytes - node_memory_Buffers_bytes - node_memory_Cached_bytes
Memory and swap memory__errors{type="correctable"}, memory__errors{type="uncorrectable"} sum by(instance)(node_edac_correctable_errors_total), sum by(instance)(node_edac_uncorrectable_errors_total)
Memory and swap swap__total, swap_total node_memory_SwapTotal_bytes
Memory and swap swap__space__free, swap_free node_memory_SwapFree_bytes
Memory and swap swap__space__used node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes
Memory and swap swap_in, swap_out (counters) node_vmstat_pswpin * 4096, node_vmstat_pswpout * 4096
Memory and swap swap__io__in, swap__io__out (per second) rate(node_vmstat_pswpin[R]) * 4096, rate(node_vmstat_pswpout[R]) * 4096
File systems bytes_total node_filesystem_size_bytes
File systems bytes_used, disk__partition__bytes_used, disk__partition__space__used node_filesystem_size_bytes - node_filesystem_free_bytes
File systems disk__partition__bytes_available, disk__partition__space__available node_filesystem_avail_bytes
File systems disk__partition__bytes_reserved, disk__partition__space__reserved node_filesystem_free_bytes - node_filesystem_avail_bytes
File systems inodes_total node_filesystem_files
File systems inodes_used, disk__partition__inodes__used node_filesystem_files - node_filesystem_files_free
File systems disk__partition__inodes__free node_filesystem_files_free
File systems disk__partition__read_write 1 - node_filesystem_readonly
Block devices disk__device__ops__read, disk__device__ops__write rate(node_disk_reads_completed_total[R]), rate(node_disk_writes_completed_total[R])
Block devices disk__device__bytes__read, disk__device__bytes__write rate(node_disk_read_bytes_total[R]), rate(node_disk_written_bytes_total[R])
Block devices disk__device__io_time__read, disk__device__io_time__write rate(node_disk_read_time_seconds_total[R]), rate(node_disk_write_time_seconds_total[R])
Block devices disk__device__io_time__total rate(node_disk_io_time_seconds_total[R]) (× 100 = utilization, %)
Network interfaces in_bytes, in_packets, in_errors, in_dropped node_network_receive_bytes_total, node_network_receive_packets_total, node_network_receive_errs_total, node_network_receive_drop_total
Network interfaces out_bytes, out_packets, out_errors, out_dropped node_network_transmit_bytes_total, node_network_transmit_packets_total, node_network_transmit_errs_total, node_network_transmit_drop_total
Network interfaces net__interface__bytes__in / __out, net__interface__packets__in / __out, net__interface__errors__in / __out, net__interface__drops__in / __out rate(<matching counter above>[R])
Network interfaces mtu node_network_mtu_bytes
Conntrack net__conntrack__count node_nf_conntrack_entries
Conntrack net__conntrack__max node_nf_conntrack_entries_limit
Netstat packets netstat__protocol__packets__in{protocol="tcp"}, {protocol="udp"}, {protocol="icmp"} node_netstat_Tcp_InSegs, node_netstat_Udp_InDatagrams, node_netstat_Icmp_InMsgs
Netstat packets netstat__protocol__packets__out{protocol="tcp"}, {protocol="udp"}, {protocol="icmp"} node_netstat_Tcp_OutSegs, node_netstat_Udp_OutDatagrams, node_netstat_Icmp_OutMsgs
Netstat packets netstat__protocol__packets__retransmits{protocol="tcp"} node_netstat_Tcp_RetransSegs
Load average load_average__1min, ___1min node_load1
Load average load_average__5min, ___5min node_load5
Load average load_average__15min, ___15min node_load15

The node_exporter netstat metrics have no protocol label; add it with label_replace(<expr>, "protocol", "tcp", "", "") if a query groups by it. Everything else the agent collects on the node — uname, time__*, raid_*, disk_smart_*, netstat__connections__*, netstat__listen__*, netstat__protocol__packets__errors, cpu_flags, cpu__physical_*, memory__numa__allocated, disk__fs__errors, disk__partition__ops__*, disk__partition__bytes__*, the text metrics fs, device, mount_options, hw_address, ipv4_addresses, the interface status status{job="net"}, net__interface__status, net__ip__status, and process metrics — is still collected under the old names.