The Prometheus monitoring module: configuration

The module does not require any configuration – it works right out-of-the-box.

The module is enabled by default in the following bundles: Default, Managed. The module is disabled by default in the Minimal bundle.

How to explicitly enable the module…

Note that the configOverrides parameter of the InitConfiguration resource is used when installing Deckhouse, not ModuleConfig.

Set the prometheusEnabled: true or prometheusEnabled: false in the configOverrides parameter to explicitly enable or disable the module when installing Deckhouse.

Use the prometheus section of the configOverrides parameter to configure the module when installing Deckhouse.

Set the spec.enabled module parameter to true or false to explicitly enable or disable the module.

Example of enabling the prometheus module:

          apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: prometheus
spec:
  enabled: true

        

Example of disabling the prometheus module:

          apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: prometheus
spec:
  enabled: false

        

The module is configured using the ModuleConfig custom resource named prometheus (learn more about setting up Deckhouse…).

Example of the ModuleConfig/prometheus resource for configuring the module:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: prometheus
spec:
  version: 2
  enabled: true
  settings: # <-- Module parameters from the "Parameters" section below.

Parameters

Schema version: 2

authobject
Options related to authentication or authorization in the application.
- auth.allowedUserGroupsarray of strings
  An array of user groups that can access Grafana & Prometheus.
  
  This parameter is used if the user-authn module is enabled or the externalAuthentication parameter is set.
  
  Caution! Note that you must add those groups to the appropriate field in the DexProvider config if this module is used together with the user-authn one.
- auth.externalAuthenticationobject
  Parameters to enable external authentication based on the Nginx Ingress external-auth mechanism that uses the Nginx auth_request module.
  
  External authentication is enabled automatically if the user-authn module is enabled.
  - auth.externalAuthentication.authSignInURLstring
    The URL to redirect the user for authentication (if the authentication service returned a non-200 HTTP response code).
    
    Example:
    authSignInURL: https://example.com/dex/sign_in
  - auth.externalAuthentication.authURLstring
    The URL of the authentication service. If the user is authenticated, the service should return an HTTP 200 response code.
    
    Example:
    authURL: https://example.com/dex/auth
- auth.satisfyAnyboolean
  Enables single authentication.
  
  If used together with the whitelistSourceRanges parameter, it authorizes all the users from above networks (no need to enter a username and password).
  
  Default: false
  Example:
```
satisfyAny: true
```
- auth.whitelistSourceRangesarray of strings
  An array if CIDRs that are allowed to authenticate in Grafana & Prometheus.
  
  Example:
```
whitelistSourceRanges:
- 1.1.1.1/32
```
grafanaobject
Grafana installation-related settings.
- grafana.customPluginsarray of strings
  A list of custom Grafana plugins. Contains plugin names from the official repository.
  
  Here is how you can add custom plugins (in this case, clickhouse-datasource and flowcharting-panel plugins are used):
  grafana: customPlugins: - agenty-flowcharting-panel - vertamedia-clickhouse-datasource
  Example:
```
customPlugins:
- agenty-flowcharting-panel
- vertamedia-clickhouse-datasource
```
- grafana.useDarkThemeboolean
  The dark theme is enabled by default.
  
  Default: false
  Example:
```
useDarkTheme: true
```
highAvailabilityboolean
Manually enable the high availability mode.

By default, Deckhouse automatically decides whether to enable the HA mode. Click here to learn more about the HA mode for modules.

Example:
```
highAvailability: true
```
httpsobject
What certificate type to use with Grafana/Prometheus.

This parameter completely overrides the global.modules.https settings.

Examples:
```
https:
  mode: CustomCertificate
  customCertificate:
    secretName: foobar
```
```
https:
  mode: CertManager
  certManager:
    clusterIssuerName: letsencrypt
```
- https.certManagerobject
  - https.certManager.clusterIssuerNamestring
    What ClusterIssuer to use for Grafana/Prometheus.
    
    Currently, letsencrypt, letsencrypt-staging, selfsigned are available. Also, you can define your own.
    
    Default: "letsencrypt"
- https.customCertificateobject
  - https.customCertificate.secretNamestring
    The name of the secret in the d8-system namespace to use with Grafana/Prometheus.
    
    This secret must have the kubernetes.io/tls format.
    
    Default: "false"
- https.modestring
  The HTTPS usage mode:
  - Disabled — Grafana/Prometheus will work over HTTP only;
  - CertManager — Grafana/Prometheus will use HTTPS and get a certificate from the clusterissuer defined in the certManager.clusterIssuerName parameter.
  - CustomCertificate — Grafana/Prometheus will use HTTPS using the certificate from the d8-system namespace.
  - OnlyInURI — Grafana/Prometheus will work over HTTP (thinking that there is an external HTTPS load balancer in front that terminates HTTPS traffic). All the links in the user-authn will be generated using the HTTPS scheme.
  Default: "Disabled"
  Allowed values: Disabled, CertManager, CustomCertificate, OnlyInURI
ingressClassstring
The class of the Ingress controller used for Grafana/Prometheus.

An optional parameter. By default, the modules.ingressClass global value is used.

Example:
```
ingressClass: public
```
longtermMaxDiskSizeGigabytesDeprecatedinteger
Deprecated and will be removed. Doesn’t affect anything. The maximum size (in GiB) to which the Longterm Prometheus’ disk can be automatically resized.

Default: 300
longtermNodeSelectorobject
The same as in the Pods’ spec.nodeSelector parameter in Kubernetes.

If the parameter is omitted or false, it will be determined automatically.

Example:
```
longtermNodeSelector:
  disktype: ssd
```
longtermRetentionDaysinteger
How long to keep the data in longterm Prometheus.

Setting this parameter to 0 will result in Longterm Prometheus not running in the cluster.

Default: 1095
longtermScrapeIntervalstring
Sets the interval for making “data snapshots” of the main Prometheus by the longterm Prometheus.

Default: "5m"
longtermStorageClassstring
The name of the storageClass to use for Longterm Prometheus.

If omitted, the StorageClass of the existing Longterm Prometheus PVC is used. If there is no PVC yet, either the global StorageClass or global.discovery.defaultStorageClass is used, and if those are undefined, the emptyDir volume is used to store the data;

CAUTION! Setting this value to one that differs from the current one (in the existing PVC) will result in Longterm Prometheus volume reprovisioning and data loss.

Example:
```
longtermStorageClass: ceph-ssd
```
longtermTolerationsarray of objects
The same as in the Pods’ spec.tolerations parameter in Kubernetes;

If the parameter is omitted or false, it will be determined automatically.

Example:
```
longtermTolerations:
- key: key1
  operator: Equal
  value: value1
  effect: NoSchedule
```
- longtermTolerations.effectstring
- longtermTolerations.keystring
- longtermTolerations.operatorstring
- longtermTolerations.tolerationSecondsinteger
- longtermTolerations.valuestring
mainMaxDiskSizeGigabytesDeprecatedinteger
Deprecated and will be removed. Doesn’t affect anything. The maximum size (in GiB) that the main Prometheus’ volume can automatically resize to.

Default: 300
nodeSelectorobject
The same as in the Pods’ spec.nodeSelector parameter in Kubernetes.

If the parameter is omitted or false, it will be determined automatically.

Example:
```
nodeSelector:
  disktype: ssd
```
retentionDaysinteger
How long to keep the data.

Default: 15
scrapeIntervalstring
Sets the interval for scraping metrics from targets.

Evaluation Interval is always equal to scrapeInterval.

Default: "30s"
storageClassstring
The name of the StorageClass to use.

If omitted, the StorageClass of the existing Prometheus PVC is used. If there is no PVC yet, either the global StorageClass or global.discovery.defaultStorageClass is used, and if those are undefined, the emptyDir volume is used to store the data.

storageClass: false — forces the emptyDir usage. You will need to delete the old PVC and restart the Pod manually.

CAUTION! Setting this value to one that differs from the current one (in the existing PVC) will result in Prometheus volume reprovisioning and data loss.

Examples:
```
storageClass: ceph-ssd
```
```
storageClass: 'false'
```
tolerationsarray of objects
The same as in the Pods’ spec.tolerations parameter in Kubernetes;

If the parameter is omitted or false, it will be determined automatically.

Example:
```
tolerations:
- key: key1
  operator: Equal
  value: value1
  effect: NoSchedule
```
- tolerations.effectstring
- tolerations.keystring
- tolerations.operatorstring
- tolerations.tolerationSecondsinteger
- tolerations.valuestring
vpaobject
Default: {"updateMode":"Initial"}
Examples:
```
vpa:
  updateMode: Initial
  longtermMaxCPU: '1'
  longtermMaxMemory: 1500Mi
  maxCPU: 1000m
  maxMemory: 1500Mi
```
```
vpa:
  updateMode: 'Off'
```
- vpa.longtermMaxCPU
  The maximum CPU requests set by the VPA controller for the Longterm Prometheus Pods.
  
  The default value is chosen automatically based on the maximum number of Pods that can be created in the cluster considering the current number of nodes and their settings. For more information, see the detect_vpa_max hook of the module.
  
  Example:
```
longtermMaxCPU: 0.1
```
- vpa.longtermMaxMemory
  The maximum Memory requests set by the VPA controller for the longterm Prometheus Pods.
  
  The default value is chosen automatically based on the maximum number of Pods that can be created in the cluster considering the current number of nodes and their settings. For more information, see the detect_vpa_max hook of the module.
  
  Example:
```
longtermMaxMemory: 4Mi
```
- vpa.maxCPU
  The maximum CPU requests set by the VPA controller for the main Prometheus Pods.
  
  The default value is chosen automatically based on the maximum number of Pods that can be created in the cluster considering the current number of nodes and their settings. For more information, see the detect_vpa_max hook of the module.
  
  Example:
```
maxCPU: '3'
```
- vpa.maxMemory
  The maximum Memory requests set by the VPA controller for the main Prometheus Pods.
  
  The default value is chosen automatically based on the maximum number of Pods that can be created in the cluster considering the current number of nodes and their settings. For more information, see the detect_vpa_max hook of the module.
  
  Example:
```
maxMemory: 3Mi
```
- vpa.updateModestring
  The Pod update mode.
  
  Default: "Initial"
  Allowed values: Initial, Auto, Off

Authentication

user-authn module provides authentication by default. Also, externalAuthentication can be configured (see below). If these options are disabled, the module will use basic auth with the auto-generated password.

Use kubectl to see password:

kubectl -n d8-system exec deploy/deckhouse -- deckhouse-controller module values prometheus -o json | jq '.prometheus.internal.auth.password'

Delete secret to re-generate password:

kubectl -n d8-monitoring delete secret/basic-auth

Note! The auth.password parameter is deprecated.

Notes

retentionSize for the main and longterm Prometheus is calculated automatically; you cannot set this value manually!
- The following calculation algorithm is used:
  - pvc_size * 0.8 — if the PVC exists;
  - 10 GiB — if there is no PVC and if the StorageClass supports resizing;
  - 25 GiB — if there is no PVC and if the StorageClass does not support resizing;
- If the local-storage is used, and you have to change the retentionSize, then you need to manually change the size of the PV and PVC. Caution! Note that the value from .status.capacity.storage PVC is used for the calculation since it reflects the actual size of the PV in the case of manual resizing.
You can change the size of Prometheus disks in the standard Kubernetes way (if the StorageClass permits this) by editing the .spec.resources.requests.storage field of the PersistentVolumeClaim resource.