Available in: Open/CE, Core, BE, SE, SE+, Ultimate/EE
Included in extensions: Billing
The module lifecycle stage: Generally available version
The module has requirements for installation
The monitoring-kubernetes module provides transparent and timely monitoring of the status of all cluster nodes and key infrastructure components.
Module features:
- provides an opportunity to plan infrastructure resources (Capacity planning);
- monitors the container runtime version (docker, containerd) on each node and checks it for compliance with the allowed versions;
- monitors the performance of the cluster monitoring subsystem itself (Dead man’s switch);
- gets metrics about the availability of file descriptors, sockets, free space, and inodes on each node;
- monitors the correct operation of key monitoring components: kube-state-metrics, node-exporter, kube-dns;
- checks the status of all nodes (
NotReady,drain,cordon) and promptly reports problems; - monitors time synchronization and notifies about deviations;
- detects cases of prolonged CPU steal overrun (when the node does not receive the required CPU time);
- controls the status of the Conntrack table on the nodes;
- shows pods with incorrect statuses, for example, if kubelet failed to do its job;
- allows you to export metrics to external monitoring systems for a single point of control.
Kubelet eviction thresholds monitoring
The module tracks how close the filesystems of every node are to the kubelet eviction thresholds. The kubelet-eviction-thresholds-exporter Deployment in the d8-monitoring namespace polls every kubelet once a minute through the Kubernetes API server (nodes/proxy):
/configz— the effectiveevictionHardandevictionSoftsettings of the kubelet. They are cached for 5 minutes; if the refresh fails, the cached values are used for up to an hour./stats/summary— the statistics of thenodefsandimagefsfilesystems (capacity, available space, inodes).
These are the same data the kubelet itself uses to decide on eviction, so the exporter does not depend on the container runtime, the disk layout or access to the node filesystem, and it works on any node OS, including Talos Linux.
Metrics
All metrics carry the node label.
| Metric | Labels | Description |
|---|---|---|
kubelet_eviction_resource_available_percent |
filesystem (nodefs, imagefs), resource (bytes, inodes) |
Share of the resource currently available, in percent of its capacity. |
kubelet_eviction_threshold_percent |
filesystem, resource, type (hard, soft) |
Eviction threshold, in percent of the resource capacity. Thresholds set in absolute units (for example, nodefs.available: "10Gi") are converted to percent using the capacity of that filesystem. The series is absent if the kubelet has no threshold for that signal. |
kubelet_eviction_node_collect_success |
— | 1 if the data of the node was collected during the last cycle, 0 otherwise. When it is 0, the other metrics of the node are absent. |
kubelet_eviction_node_config_refresh_success |
— | 1 if the thresholds come from a recently fetched kubelet configuration, 0 if the cached configuration is used because refreshing it failed. |
The imagefs metrics are absent for a node if the kubelet does not report a separate image filesystem in /stats/summary.
The KubeletNodeFSBytesUsage, KubeletNodeFSInodesUsage, KubeletImageFSBytesUsage and KubeletImageFSInodesUsage alerts compare kubelet_eviction_resource_available_percent with kubelet_eviction_threshold_percent and fire as the available share approaches the soft threshold, approaches the hard threshold, drops below it, and reaches zero.
Requirements
The exporter reads the kubelet configuration from the /configz endpoint, which the kubelet serves only when enableDebuggingHandlers is enabled (the default; kubectl logs and kubectl exec depend on the same setting). If it is disabled on a node, that node exposes only kubelet_eviction_node_collect_success 0.
Migration from earlier versions
Earlier versions of the module exported the eviction thresholds from a sidecar container of node-exporter under different names, with a mountpoint label and without the available share:
| Removed metric | Replacement |
|---|---|
kubelet_eviction_nodefs_bytes{type="hard"|"soft"} |
kubelet_eviction_threshold_percent{filesystem="nodefs",resource="bytes",type="hard"|"soft"} |
kubelet_eviction_nodefs_inodes{type="hard"|"soft"} |
kubelet_eviction_threshold_percent{filesystem="nodefs",resource="inodes",type="hard"|"soft"} |
kubelet_eviction_imagefs_bytes{type="hard"|"soft"} |
kubelet_eviction_threshold_percent{filesystem="imagefs",resource="bytes",type="hard"|"soft"} |
kubelet_eviction_imagefs_inodes{type="hard"|"soft"} |
kubelet_eviction_threshold_percent{filesystem="imagefs",resource="inodes",type="hard"|"soft"} |
The mountpoint label is no longer available: the exporter reports what the kubelet considers nodefs and imagefs rather than a mount point. Queries that combined the old metrics with node_filesystem_avail_bytes or node_filesystem_files_free by mountpoint can use kubelet_eviction_resource_available_percent directly. For example, the available share of nodefs compared with the hard threshold:
kubelet_eviction_resource_available_percent{filesystem="nodefs", resource="bytes"}
<
on (node, filesystem, resource)
kubelet_eviction_threshold_percent{filesystem="nodefs", resource="bytes", type="hard"}