Recording rules

Recording rules are calculated from other metrics using PromQL expressions and stored as separate time series.

Typical use cases include:

• Complex or resource-intensive queries.
• Queries that run frequently (for example, in multiple dashboards or alerts).
• Queries that create additional load on the metric storage.

Recording rules are defined in rule groups as items in the spec.rules array. If a rule contains the record field, it is treated as a recording rule.

Types of rule groups with derived metrics

Three types of rule groups can define derived metrics:

Rule group types:

• System rule groups (ClusterObservabilityMetricsRulesGroup): Used to calculate derived metrics for platform-level and cluster component metrics. Created and managed by DKP administrators.

• Project rule groups (ObservabilityMetricsRulesGroup): Used to calculate derived metrics within a specific project (namespace). Project users can create and edit them within the configured access permissions.

• Standard propagated rule groups (ClusterObservabilityPropagatedMetricsRulesGroup): Created at the cluster level and automatically become available in all projects.

Recording rules from logs (LogQL)

In addition to metric-based recording rules, the module can derive recording rules from logs using LogQL. These rules are evaluated by the logs-ruler service, which makes LogQL queries to the cluster log storage backend, and the resulting series are exposed as regular Prometheus metrics.

Rule group types for log-based rules:

Only recording rules are supported (the record and expr fields).

Log-based recording rules can be used as any other Prometheus metrics, including alert configuration using the ObservabilityMetricsRulesGroup resource.

For project-scope rule groups, LogQL expressions are automatically restricted by the resource namespace. Do not use the namespace label in stream selectors since the module adds it automatically based on the current project.

Each recorded series gets the following platform labels: scope_type (namespaced or cluster), created_by=observability, rule_group_name, and namespace (for project scope). These labels are reserved and cannot be set by users.

Example

In the following example, LogQL derives and writes a log-based recording rule. After that, an alerting rule is created based on it.

1. Writing a log-based recording rule (project scope):

   apiVersion: observability.deckhouse.io/v1alpha1
   kind: ObservabilityLogsRulesGroup
   metadata:
     name: app-errors
     namespace: my-project
   spec:
     interval: 1m
     rules:
       - record: app_log_errors_per_second
         expr: 'sum(rate({app="my-app"} |= "error" [5m]))'

2. Creating an alerting rule for the recorded metric via a rule group:

   apiVersion: observability.deckhouse.io/v1alpha1
   kind: ObservabilityMetricsRulesGroup
   metadata:
     name: app-errors-alerts
     namespace: my-project
   spec:
     rules:
       - alert: AppLogErrorsHigh
         expr: 'app_log_errors_per_second > 1'
         for: 5m
         labels:
           severity: S5
         annotations:
           summary: "High log error rate in my-app"