The extended-monitoring module

The extended-monitoring module extends cluster monitoring capabilities with additional Prometheus exporters, which allow you to identify potential problems before they affect the operation of services.

Module features:

• Advanced Metrics Collection — collects additional metrics, and also includes ready-made alerts and dashboards that allow you to detect and diagnose incidents faster:
  • collects and exports metrics for free space and inodes on nodes, as well as for objects with a label extended-monitoring.deckhouse.io/enabled="" in the namespace;
  • automatically generates alerts when the thresholds are reached.
• Container image monitoring:
  • adds metrics and sends alerts about unavailability of container images to registry for all types of workload (Deployments, StatefulSets, DaemonSets, CronJobs);
  • helps to find out in advance about possible problems with launching or updating pods.
• Cluster Events — collects Kubernetes events and displays them as metrics, which allows you to track the dynamics of changes and respond faster to incidents.
• Certificate control:
  • scans the cluster’s Secrets and generates metrics about the expiration of x509 certificates;
  • allows you not to miss critical moments and update certificates on time, avoiding application downtime due to expired certificates.

How it works

The module exports specific Kubernetes object labels to Prometheus. This lets you set alert trigger thresholds via labels instead of hard-coded numbers in Prometheus rules.

Before (the threshold 1 is hard-coded in the rule):

(
  kube_statefulset_status_replicas - kube_statefulset_status_replicas_ready
)
> 1

After (the threshold comes from a module metric and can be set with a label):

(
  kube_statefulset_status_replicas - kube_statefulset_status_replicas_ready
)
> on (namespace, statefulset)
(
  max by (namespace, statefulset) (extended_monitoring_statefulset_threshold{threshold="replicas-not-ready"})
)

Usage

Attach the extended-monitoring.deckhouse.io/enabled label to the Namespace to enable the export of extended monitoring metrics. You can do it, for example:

• in the application Helm chart — set the label in the Namespace manifest (recommended for GitOps);
• in a CI/CD pipeline, for example in .gitlab-ci.yml, with d8 k label namespace <NAMESPACE_NAME> extended-monitoring.deckhouse.io/enabled="";
• manually: d8 k label namespace my-app-production extended-monitoring.deckhouse.io/enabled="";
• centrally via the namespace-configurator module, if you need to apply the label to multiple namespaces at once.

where <NAMESPACE_NAME>: Name of the Kubernetes namespace.

After that, metrics and alert trigger thresholds with default values appear in Prometheus for all supported Kubernetes objects in the namespace. Monitoring is also enabled automatically for a number of non-namespaced Kubernetes objects described below.

Overriding an alert trigger threshold for an object

To override an alert trigger threshold for an object, add a label with the threshold.extended-monitoring.deckhouse.io/ prefix and a threshold name from the list below. For example:

d8 k label pod test threshold.extended-monitoring.deckhouse.io/disk-inodes-warning=30

In this case, the disk-inodes-warning alert trigger threshold for the test pod becomes 30 instead of the default.

Overriding alert trigger thresholds for a namespace

To override alert trigger thresholds for a namespace, set the same label at the namespace level. For example:

d8 k label namespace my-app-production threshold.extended-monitoring.deckhouse.io/5xx-warning=20

This replaces the default threshold for all objects in the namespace that do not already have this label on the object itself.

Disabling monitoring for a specific object

To disable monitoring for a specific object, add the extended-monitoring.deckhouse.io/enabled=false label to the object. This disables the default threshold labels and the alerts tied to them.

Standard labels and supported Kubernetes objects

Below is the list of labels used in Prometheus Rules and their default values.

For example, the label threshold.extended-monitoring.deckhouse.io/5xx-warning: "5" on the Ingress resource changes the alert threshold from 10% (default) to 5%.

Non-namespaced Kubernetes objects

Non-namespaced Kubernetes objects do not need labels on the namespace, and monitoring on them is enabled by default when the module is enabled.

Node

Namespaced Kubernetes objects

Pod

Ingress

Deployment

The threshold implies the number of unavailable replicas in addition to maxUnavailable. This threshold will be triggered if the number of unavailable replicas is greater than maxUnavailable by the amount specified. Suppose replicas-not-ready is 0. In this case, the threshold will be triggered if the number of unavailable replicas is greater than maxUnavailable. If replicas-not-ready is set to 1, then the threshold will be triggered if the number of unavailable replicas is greater than maxUnavailable + 1. This way, you can fine-tune this parameter for specific Deployments (that may be unavailable) in the namespace with the extended monitoring enabled to avoid getting excessive alerts.

StatefulSet

The threshold implies the number of unavailable replicas in addition to maxUnavailable (comments on Deployment).

DaemonSet

The threshold implies the number of unavailable replicas in addition to maxUnavailable (comments on Deployment).

CronJob

Note that only the deactivation using the extended-monitoring.deckhouse.io/enabled=false label is supported.