Metrics

The observability module provides access control mechanisms for cluster-level and project-level metrics, as well as tools for interactive metric analysis.

Depending on the scope, two types of metrics are distinguished:

The module allows running arbitrary PromQL queries against cluster or project metrics directly from the Deckhouse web UI.

Interactive metric analysis capabilities:

• Running arbitrary PromQL queries against metrics.
• Selecting the data source (main Prometheus, Longterm, and custom sources).
• Visualizing results as tables and graphs.
• Metric and label autocomplete while typing queries.

Available data exploration modes:

• System data exploration: Available to DKP administrators in the Deckhouse web UI under “System” → “System management” → “Monitoring” → “Data exploration”. This section provides access to cluster metrics.

• Project data exploration: Available to DKP project users in the Deckhouse web UI in the corresponding project under “Monitoring” → “Data exploration”. In this section, queries are limited to metrics related to the current project (within its namespace).

External access to metrics

The module allows configuring external access to cluster metrics for integration with external monitoring systems (for example, external Grafana) or for writing metrics from external sources.

To enable external access, use the externalMetricsAccess parameter. Request authorization is performed using a service account with read or write permissions for metrics.

The following scenarios are supported:

• Reading cluster metrics: Access to system metrics across the entire cluster.
• Reading project metrics: Access to metrics of a specific namespace through a Prometheus-compatible API.
• Writing metrics: Sending metrics from external sources using the Prometheus Remote Write protocol.

External access for reading cluster metrics

To provide external access to cluster metrics, follow these steps:

1. Enable external access to metrics. To do this, enable the externalMetricsAccess parameter.

2. Create a ServiceAccount for request authorization:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: cluster-metrics-access
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: cluster-metrics-access
     annotations:
       kubernetes.io/service-account.name: cluster-metrics-access
   type: kubernetes.io/service-account-token

3. Grant read permissions for metrics to the created ServiceAccount using ClusterRole and ClusterRoleBinding:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: observability-cluster-metrics-viewer
   rules:
     - apiGroups: ["observability.deckhouse.io"]
       resources: ["clustermetrics"]
       verbs: ["get", "list", "watch"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: bind-observability-cluster-metrics-viewer
   subjects:
     - kind: ServiceAccount
       name: cluster-metrics-access
       namespace: default
   roleRef:
     kind: ClusterRole
     name: observability-cluster-metrics-viewer
     apiGroup: rbac.authorization.k8s.io

4. Retrieve the authorization token for metric access.

   When the ServiceAccount was created, a secret containing the token was also created. The token is stored as a Base64-encoded string. To extract and decode it, run the following command:

   d8 k -n my-namespace get secret cluster-metrics-access -ojsonpath='{ .data.token }' | base64 -d

   This token will be required in the next step when configuring a Grafana data source.

5. Configure metric access in external Grafana.

   Add a Prometheus data source with the following parameters:

   • Name: Any data source name.
   • URL: External endpoint address for metric access in the format https://observability.%publicDomainTemplate%/<prefix>, where:
     • %publicDomainTemplate%: DNS name template of your cluster configured in DKP global settings.
     • <prefix>: One of the Prometheus access prefixes:
       • /metrics/main: For main Prometheus metrics.
       • /metrics/longterm: For Prometheus Longterm metrics.
   • HTTP Headers: Additional HTTP headers for authorization:
     • Header: Authorization.
     • Value: Bearer <TOKEN_VALUE>, where <TOKEN_VALUE> is the token obtained from the secret in the previous step.

External access for reading project metrics

To provide external access to project metrics, follow these steps:

1. Enable external access to metrics. To do this, enable the externalMetricsAccess module parameter.

2. Create a ServiceAccount for request authorization:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: metrics-access
     namespace: my-namespace
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: metrics-access
     annotations:
       kubernetes.io/service-account.name: metrics-access
   type: kubernetes.io/service-account-token

3. Grant read permissions for metrics to the created ServiceAccount using Role and RoleBinding:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     namespace: my-namespace
     name: metrics-access
   rules:
     - apiGroups: ["observability.deckhouse.io"]
       resources: ["metrics"]
       verbs: ["get", "watch", "list"]
     - apiGroups: [""]
       resources: ["namespaces"]
       verbs: ["get", "watch", "list"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: metrics-access
     namespace: my-namespace
   subjects:
     - kind: ServiceAccount
       name: metrics-access
       namespace: my-namespace
   roleRef:
     kind: Role
     name: metrics-access
     apiGroup: rbac.authorization.k8s.io

4. Retrieve the authorization token for metric access.

   When the ServiceAccount was created, a secret containing the token was also created. The token is stored as a Base64-encoded string. To extract and decode it, run the following command:

   d8 k -n my-namespace get secret metrics-access -ojsonpath='{ .data.token }' | base64 -d

   This token will be required in the next step when configuring a Grafana data source.

5. Configure metric access in external Grafana.

   Add a Prometheus data source with the following parameters:

   • Name: Any data source name.
   • URL: External endpoint address for metric access in the format https://observability.%publicDomainTemplate%/<prefix>, where:
     • %publicDomainTemplate%: DNS name template of your cluster configured in DKP global settings.
     • <prefix>: One of the Prometheus access prefixes:
       • /metrics/main: For main Prometheus metrics.
       • /metrics/longterm: For Prometheus Longterm metrics.
   • HTTP Headers: Additional HTTP headers for authorization:
     • Header: Authorization.
     • Value: Bearer <TOKEN_VALUE>, where <TOKEN_VALUE> is the token obtained from the secret in the previous step.

External access for writing metrics

To provide external access for writing metrics, follow these steps:

1. Enable external access to metrics. To do this, enable the externalMetricsAccess module parameter.

2. Create a ServiceAccount for request authorization:

   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: metrics-access
     namespace: my-namespace
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: metrics-access
     annotations:
       kubernetes.io/service-account.name: metrics-access
   type: kubernetes.io/service-account-token

3. Grant write permissions for metrics to the created ServiceAccount using Role and RoleBinding:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     namespace: my-namespace
     name: metrics-access
   rules:
     - apiGroups: ["observability.deckhouse.io"]
       resources: ["metrics"]
       verbs: ["create"]
     - apiGroups: [""]
       resources: ["namespaces"]
       verbs: ["get", "watch", "list"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: metrics-access
     namespace: my-namespace
   subjects:
     - kind: ServiceAccount
       name: metrics-access
       namespace: my-namespace
   roleRef:
     kind: Role
     name: metrics-access
     apiGroup: rbac.authorization.k8s.io

4. Retrieve the authorization token for writing metrics.

   When the ServiceAccount was created, a secret containing the token was also created. The token is stored as a Base64-encoded string. To extract and decode it, run the following command:

   d8 k -n my-namespace get secret metrics-access -ojsonpath='{ .data.token }' | base64 -d

5. To write metrics, send requests using Prometheus Remote Write V1 or V2 with the following parameters:

   • URL: External endpoint address for writing metrics in the format https://observability.%publicDomainTemplate%/api/v1/write, where:
     • %publicDomainTemplate%: DNS name template of your cluster configured in DKP global settings.
   • HTTP Headers: Additional HTTP headers for authorization:
     • Header: Authorization.
     • Value: Bearer <TOKEN_VALUE>, where <TOKEN_VALUE> is the token obtained from the secret in the previous step.