Available in:  Open/CE, Core, BE, SE, SE+, Ultimate/EE

The module lifecycle stage: Generally available version

The module has requirements for installation

How to explicitly enable or disable the module…

You may explicitly enable or disable the module in one of the following ways:

  • Via Deckhouse web UI. In the “System” → “System Management” → “Deckhouse” → “Modules” section, open the observability module and enable (or disable) the “Module enabled” toggle. Save changes.

    Example:

    Module enable/disable interface
  • Via Deckhouse CLI (d8).

    Use the d8 system module enable command for enabling, or d8 system module disable command for disabling the module (you need Deckhouse CLI (d8), configured to work with the cluster).

    Example of enabling the module:

    d8 system module enable observability
  • Using ModuleConfig observability.

    Set spec.enabled to true or false in ModuleConfig observability (create it if necessary).

    Example of a manifest to enable module observability:

    apiVersion: deckhouse.io/v1alpha1
    kind: ModuleConfig
    metadata:
      name: observability
    spec:
      enabled: true

How to configure the module…

You can configure the module in one of the following ways:

  • Via Deckhouse web UI.

    In the “System” → “System Management” → “Deckhouse” → “Modules” section, open the observability module and enable the “Advanced Settings” toggle. Fill in the required fields in the “Configuration” tab or specify the module settings in YAML format on the “YAML” tab, without the top-level settings key. Save the changes.

    Example:

    Module settings interface

    You can also edit the ModuleConfig object observability on the “YAML” tab in the module settings window (“System” → “System Management” → “Deckhouse” → “Modules”, open the module observability) by specifying the schema version in the spec.version parameter and the necessary module parameters in the spec.settings section.

  • Via Deckhouse CLI (d8) (requires Deckhouse CLI (d8) configured to work with the cluster).

    Edit the existing ModuleConfig observability (for more details on configuring Deckhouse Platform, see the documentation) by executing the following command:

    d8 k edit mc observability

    Make the necessary changes in the spec.settings section. If necessary, specify the schema version in the spec.version parameter. Save the changes.

    You can also create a manifest file for ModuleConfig observability using the example below. Fill in the spec.settings section with the required module parameters. If necessary, specify the schema version in the spec.version parameter.

    Apply the manifest using the following command (indicate the manifest file name):

    d8 k apply -f <FILENAME>

    Example of a manifest for ModuleConfig observability:

    apiVersion: deckhouse.io/v1alpha1
    kind: ModuleConfig
    metadata:
      name: observability
    spec:
      version: 2
      enabled: true
      settings: # Module parameters from the "Parameters" section below.

How to change the module release channel…

To change the module release channel, follow the steps in How do I change the release channel for a module?.

Requirements

To the Deckhouse Platform version: 1.74 and above.

To the versions of other modules:

  • console: 1.41.1 and above.
  • operator-prometheus: any version.
  • prometheus: any version.

Conversions

The module is configured using the ModuleConfig resource, the schema of which contains a version number. When you apply an old version of the ModuleConfig schema in a cluster, automatic transformations are performed. To manually update the ModuleConfig schema version, the following steps must be completed sequentially for each version:

  • Updates from version 1 to 2:

    Delete the .dop.nodeAffinity field: the agent collects node metrics instead of node-exporter and runs on every node of the cluster, restricting the set of nodes is no longer supported.

Parameters

Schema version: 2

  • settings
    object
    • settings.deadMansSwitch
      object
      Settings for the DeadMansSwitch alerting.

      Default: {}

      • settings.deadMansSwitch.enabled
        boolean

        Whether to enable DeadMansSwitch alert handling.

        When enabled (default), a DeadMansSwitch heartbeat alert is expected from Prometheus and a PrometheusUnavailable alert is generated if the heartbeat is missing. When disabled, neither DeadMansSwitch nor PrometheusUnavailable alerts are created.

        Default: true

    • settings.dop
      object
      Settings for integration with DOP (Deckhouse Observability Platform).

      Default: {}

      • settings.dop.destinations
        array of objects
        List of DOP destinations for metrics delivery. At least one destination is required.

        Example:

        destinations:
        - host: dop.flant.com
          token: your-api-token-here
        
        • settings.dop.destinations.host
          string
          DOP platform host address (must be a valid FQDN, e.g., dop.flant.com).

          Length: 1..253

          Pattern: ^[a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(\.[a-zA-Z0-9]([a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$

        • settings.dop.destinations.token
          string
          API token for the DOP platform (UUID format).

          Pattern: ^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$

      • settings.dop.hostMetrics
        boolean
        Whether to enable nodes metrics collection with okagent.

        Default: true

    • settings.externalMetricsAccess
      object
      Settings for external access to metrics.

      Default: {}

      • settings.externalMetricsAccess.enabled
        boolean

        Whether to enable metrics external read access.

        If enabled, the observability module will expose the metrics via ingress for reading. Metrics can be read using the Prometheus-compatible API. Providing a valid ServiceAccount token via the Authorization: Bearer header is mandatory. For details, refer to “External access to metrics”.

        Default: false

      • settings.externalMetricsAccess.whitelistSourceRanges
        array of strings
        An array if CIDRs that are allowed to read metrics. If not specified, no restrictions are applied.

        Default: []

        Example:

        whitelistSourceRanges:
        - 1.1.1.1/32
        
        • settings.externalMetricsAccess.whitelistSourceRanges.Element of the array
          string

          Pattern: ^(((25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9]{2}|[1-9]?[0-9])(/([0-9]|[12][0-9]|3[0-2]))?|([0-9A-Fa-f]{0,4}:){2,7}[0-9A-Fa-f]{0,4}(/([0-9]|[1-9][0-9]|1[01][0-9]|12[0-8]))?)$

    • settings.grafana
      object
      Grafana related settings.

      Default: {}

      • settings.grafana.customPlugins
        array of strings

        A list of custom Grafana plugins. Contains plugin names from the official repository.

        Here is how you can add custom plugins (in this case, clickhouse-datasource and flowcharting-panel plugins are used):

        grafana:
          customPlugins:
          - agenty-flowcharting-panel
          - vertamedia-clickhouse-datasource

        You can also install plugins from other sources by passing a link to the plugin zip archive in the format <url to plugin zip>;<plugin name>:

        grafana:
          customPlugins:
          - http://10.241.32.16:3000/netsage-bumpchart-panel-1.1.1.zip;netsage-bumpchart-panel

        Example:

        customPlugins:
        - agenty-flowcharting-panel
        - vertamedia-clickhouse-datasource
        
    • settings.highAvailability
      boolean

      Manually enable the high availability mode.

      By default, the mode is taken from the global highAvailability parameter; if it is not set, the mode is enabled when the cluster control plane is highly available.

      alert-kube-api and observability-webhook always run in two replicas regardless of this parameter.

      Examples:

      highAvailability: true
      
      highAvailability: false
      
    • settings.https
      object

      What certificate type to use with observability module.

      This parameter completely overrides the global.modules.https settings.

      Examples:

      customCertificate:
        secretName: foobar
      mode: CustomCertificate
      
      certManager:
        clusterIssuerName: letsencrypt
      mode: CertManager
      
      • settings.https.certManager
        object
        • settings.https.certManager.clusterIssuerName
          string

          What ClusterIssuer to use for the observability module.

          Currently, letsencrypt, letsencrypt-staging, selfsigned are available. Also, you can define your own.

          Default: letsencrypt

      • settings.https.customCertificate
        object

        Default: {}

        • settings.https.customCertificate.secretName
          string

          The name of the secret in the d8-system namespace to use with the observability module.

          This secret must have the kubernetes.io/tls format.

          Default: false

      • settings.https.mode
        string

        The HTTPS usage mode:

        • Disabled — the observability module will work over HTTP only;
        • CertManager — the observability module will use HTTPS and get a certificate from the clusterissuer defined in the certManager.clusterIssuerName parameter.
        • CustomCertificate — the observability module will use HTTPS using the certificate from the d8-system namespace.
        • OnlyInURI — the observability module will work over HTTP (thinking that there is an external HTTPS load balancer in front that terminates HTTPS traffic). All the links in the user-authn will be generated using the HTTPS scheme. Load balancer should provide a redirect from HTTP to HTTPS.

        Default: Disabled

        Allowed values: Disabled, CertManager, CustomCertificate, OnlyInURI

    • settings.ingressClass
      string

      The ingress class to use for all ingress resources created by the observability module.

      This parameter completely overrides the global.modules.ingressClass settings. If not specified, the global ingress class configuration will be used.

      The value must be a valid Kubernetes resource name (lowercase alphanumeric characters or hyphens, must start and end with alphanumeric character).

      Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$

      Examples:

      ingressClass: nginx
      
      ingressClass: nginx-external
      
    • settings.lightweightLogs
      object
      Settings for lightweight logs collection directly from kubelet.

      Default: { "enabled": true }

      • settings.lightweightLogs.enabled
        boolean
        Whether to enable lightweight logs collection from kubelet.

        Default: true

    • settings.opagent
      object
      Settings for the opagent host metrics collector.

      Default: {}

      • settings.opagent.scrapeInterval
        integer

        Interval (in seconds) at which opagent collects host metrics.

        Should typically match the cluster’s Prometheus scrape_interval to avoid mismatched data density in Grafana dashboards backed by the same datasource.

        Default: 30

        Allowed values: 10 <= X

    • settings.polkAnnotations
      object
      Settings for Polk-specific alert annotations handling.

      Default: {}

      • settings.polkAnnotations.enabled
        boolean

        Whether to keep Polk-specific alert annotations (with the plk_ prefix) in incoming alerts.

        When enabled (default), plk_* annotations are preserved in alerts for all notification receivers. When disabled, all plk_* annotations are removed from alerts when sending notifications to any receiver.

        Default: true