DP cluster management

The commander and commander-agent modules are used to implement cluster management in Deckhouse Platform (DP).

A DP cluster with the commander module installed is a managing cluster. A DP cluster with the commander-agent module installed is a managed cluster.

Сommander module

The commander module implements a web application that allows you to create standardized DP clusters and manage their configurations and lifecycle.

To install the commander module, you need a PostgreSQL instance. It can be deployed inside or outside the cluster by using the managed-postgresql module.

For details about features, configuration, and usage examples, see the module documentation section.

Сommander module architecture

The Level 2 C4 architecture of the commander module and its interaction with other DP components are shown in the following diagram:

Сommander module components

The module consists of core, service, and optional components.

Core components

Core components of the commander module include:

1. Frontend (Deployment): Provides a web interface for cluster management.

   Contains the following containers:

   • wait-postgres: An init container that checks PostgreSQL instance availability.
   • wait-migrations: An init container that waits for all database migrations to complete.
   • frontend: The main container.

2. Backend (Deployment): The main component that implements the following APIs:

   • UI API: Manages cluster, template, and workspace settings.
   • External API for integrations and automation: Operates on clusters, tasks, templates, projects, roles, and more.
   • API for cluster-manager: Exchanges cluster task states and processes deployment and configuration phases of managed clusters.
   • API for commander-agent: Authorizes and receives agent metrics, and provides target configuration for a managed cluster.

   Backend provides the following capabilities:

   • Cluster lifecycle management: Creates, modifies, or deletes a cluster (through cluster-manager), restarts failed tasks, and runs related checks.
   • Cluster RBAC management and API operation access restrictions.
   • Template management and storage of target and current cluster settings.
   • Container registry management.
   • Project and project template management.
   • Billing settings management (if this component is enabled).

   The component stores the target system state in a PostgreSQL instance. It also uses a Redis instance to publish tasks to managed clusters and receive task processing results.

   Contains the following containers:

   • wait-postgres: An init container that checks PostgreSQL instance availability.
   • wait-migrations: An init container that waits for all database migrations to complete.
   • backend: The main container.
   • kube-rbac-proxy: A sidecar container with a Kubernetes RBAC-based authorization proxy that provides secure access to the main container.

3. Cable (Deployment): Provides WebSocket support for real-time UI updates.

   Contains the following containers:

   • wait-postgres: An init container that checks PostgreSQL instance availability.
   • wait-migrations: An init container that waits for all database migrations to complete.
   • cable: The main container.
   • kube-rbac-proxy: A sidecar container with a Kubernetes RBAC-based authorization proxy that provides secure access to the main container.

4. Sidekiq (Deployment): A component based on the Sidekiq library that provides queue processing in Ruby on Rails.

   Contains the following containers:

   • wait-postgres: An init container that checks PostgreSQL instance availability.
   • wait-migrations: An init container that waits for all database migrations to complete.
   • sidekiq: The main container.
   • kube-rbac-proxy: A sidecar container with a Kubernetes RBAC-based authorization proxy that provides secure access to the main container.

5. Cluster-manager (Deployment): Executes tasks requested by backend to deploy managed DP clusters and manage their configuration.

   When a new cluster is created, cluster-manager creates the following resources in the d8-commander namespace of the managing cluster:

   • Deployment and Secret named dhctl-<REGISTRY_HASH>-<VERSION>: Runs dhctl in gRPC server mode. Cluster-manager uses RPC calls to install and configure a DP cluster.
   • To process incoming commander-agent connections and implement a reverse interaction channel with the control plane of the corresponding cluster:
     • Deployment (ampg-connector-<UUID>)
     • Service:
       • ampg-connector-<UUID>
       • ampg-console-backend-<UUID>
       • ampg-api-server-<UUID>
       • ampg-deckhouse-tools-<UUID>
       • ampg-aggregating-proxy-<UUID>
       • ampg-upmeter-<UUID>
       • ampg-label-proxy-<UUID>
     • Secret (ampg-console-backend-<UUID>)
     • Ingress (ampg-agent-api-connector-<UUID>, ampg-console-backend-<UUID>)
   • DexClient (cluster-<UUID>): Configures the authentication service for a managed cluster using the capabilities of the user-authn module.

   During cluster bootstrap, cluster-manager also performs the following actions in the managed cluster:

   • Installs the commander-agent module.
   • Disables the terraform-manager module.
   • Configures authentication based on the Dex controller of the user-authn module. As a Dex provider, cluster-manager sets cluster-<UUID> DexClient from the managing cluster.
   • Configures the managed cluster connection to a container registry.
   • If billing is enabled, creates a PrometheusRemoteWrite custom resource to send metrics to the billing-prometheus component in the managing cluster.

   Contains the following containers:

   • wait-postgres: An init container that checks PostgreSQL instance availability.
   • wait-migrations: An init container that waits for all database migrations to complete.
   • cluster-manager: The main container.
   • kube-rbac-proxy: A sidecar container with a Kubernetes RBAC-based authorization proxy that provides secure access to the main container.

6. Dhctl-<REGISTRY_HASH>-<VERSION> (Deployment): A component with a single dhctl container. It runs the dhctl utility in gRPC server mode. Cluster-manager uses RPC calls to the dhctl server to install or configure a managed DP cluster.

7. Ampg-connector-<UUID> (Deployment): A component with a single main container that establishes a tunnel and proxies requests to the control plane of a managed DP cluster through the incoming commander-agent connection.

   The component is created by cluster-manager for each managed DP cluster.

8. Cluster-checker (Deployment): Periodically starts cluster verification tasks.

   Contains the following containers:

   • wait-postgres: An init container that checks PostgreSQL instance availability.
   • wait-migrations: An init container that waits for all database migrations to complete.
   • backend: The main container.

9. Cluster-task-checker (Deployment): Periodically searches for running tasks that have not reported their status for a long time and moves them to lost or lost_in_critical.

   Contains the following containers:

   • wait-postgres: An init container that checks PostgreSQL instance availability.
   • wait-migrations: An init container that waits for all database migrations to complete.
   • backend: The main container.

10. Redis (Deployment): A component with a single redis container that provides a dedicated Redis database instance for storing task queue data and commander session states.

11. Console-frontend (Deployment): Provides the managed DP cluster administration web interface.

    Contains the following containers:

    • wait-postgres: An init container that checks PostgreSQL instance availability.
    • wait-migrations: An init container that waits for all database migrations to complete.
    • nginx: The main container.

12. Console-backend (Deployment): An administration API backend for the managed DP cluster that serves requests from the cluster-manager component.

    Console-backend uses Secret and Service resources created by cluster-manager to connect to the control plane of a managed DP cluster.

    Contains the following containers:

    • wait-postgres: An init container that checks PostgreSQL instance availability.
    • wait-migrations: An init container that waits for all database migrations to complete.
    • backend: The main container.

Service components

Service components of the commander module include:

1. Cluster-task-cleaner (CronJob): Runs once a day and removes completed tasks created more than 30 days ago.

   Contains the following containers:

   • wait-postgres: An init container that checks PostgreSQL instance availability.
   • wait-migrations: An init container that waits for all database migrations to complete.
   • cluster-task-cleaner: The main container.

2. Agent-tokens-rotation (CronJob): Regularly rotates commander-agent tokens.

   Contains the following containers:

   • wait-postgres: An init container that checks PostgreSQL instance availability.
   • wait-migrations: An init container that waits for all database migrations to complete.
   • agent-tokens-rotation: The main container.

3. Encryption-migrator (CronJob): Runs once a day and migrates encrypted data from SHA-1 to SHA-256. Not all data may be processed in a single run. In addition, after restoring from backup or due to delayed migrations, SHA-1 records may appear.

   Contains the following containers:

   • wait-postgres: An init container that checks PostgreSQL instance availability.
   • wait-migrations: An init container that waits for all database migrations to complete.
   • encryption-migrator: The main container.

4. Audits-cleaner (CronJob): Runs once a day and deletes old audit logs.

   The Deckhouse controller of the deckhouse module creates audits-cleaner if the settings.featureFlags.auditsRetentionDays module parameter is set and defines the audit log retention period.

   Contains the following containers:

   • wait-postgres: An init container that checks PostgreSQL instance availability.
   • wait-migrations: An init container that waits for all database migrations to complete.
   • audits-cleaner: The main container.

Optional components

Optional components of the commander module include:

1. Billing-prometheus (StatefulSet): Runs Deckhouse Prom++ in metrics ingestion mode over the Prometheus Remote Write protocol.

   The prometheus module in a managed DP cluster sends metrics. The component stores resource usage history for all managed clusters, which is an important data source for billing reports.

   Contains the following containers:

   • prometheus: The main container.
   • kube-rbac-proxy: A sidecar container with a Kubernetes RBAC-based authorization proxy that provides secure access to the main container.

2. Billing-reports (StatefulSet): A component that runs the nginx web server as billing report storage. Reports are uploaded to nginx over Web-based Distributed Authoring and Versioning (WebDAV).

   Contains the following containers:

   • nginx: The main container.
   • kube-rbac-proxy: A sidecar container with a Kubernetes RBAC-based authorization proxy that provides secure access to the main container.

3. Billing-schedules-reporter (Deployment): Implements a scheduler that starts and controls tasks for generating required billing reports.

   The billing-schedules-reporter component saves generated reports to billing-reports over WebDAV.

   Contains the following containers:

   • wait-postgres: An init container that checks PostgreSQL instance availability.
   • wait-migrations: An init container that waits for all database migrations to complete.
   • backend: The main container.

4. Billing-aggregating-proxy (Deployment): Aggregates PromQL queries, proxies them to billing-prometheus, and returns query results.

   Contains the following containers:

   • billing-aggregating-proxy: A sidecar container based on Grafana Mimir. It provides query optimization and data caching. If data is missing in the cache, Grafana Mimir forwards the request to billing-prometheus via the promxy sidecar container.
   • promxy: A sidecar container based on Promxy. It proxies requests to billing-prometheus and provides a single endpoint for accessing data from multiple Deckhouse Prom++ instances.
   • kube-rbac-proxy: A sidecar container with a Kubernetes RBAC-based authorization proxy that provides secure access to the main container.

Module interactions

The module interacts with the following components:

1. Kube-apiserver:

   • Works with Deployment, Service, Secret, and Ingress resources in the d8-commander namespace.
   • Authorizes requests.

2. Container registry: Retrieves available DP releases by release channels.

3. PostgreSQL instance: Stores cluster states and metadata, tasks, tokens, projects, billing parameters, and reports.

4. Managed cluster: Creates, modifies, or deletes DP clusters.

The following external components interact with the module:

• Controller-nginx: Forwards external requests to module component endpoints.

• Prometheus-main: Collects metrics from module components.

Сommander-agent module

The commander-agent module establishes a service connection to a managing DP cluster.

For details about module settings, see the module documentation section.

Сommander-agent module architecture

The Level 2 C4 architecture of the commander-agent module and its interaction with other DP components are shown in the following diagram:

Сommander-agent module components

The module consists of one agent component and one commander-agent container.

Agent performs the following actions:

• Brings managed cluster configuration in line with target configuration received from commander module backend.
• Collects managed cluster metrics:
  • CPU cores: Number of node CPU cores.
  • Memory: Total node memory in GiB.
  • Filesystem: Total size of node root disks in GiB.
  • PVC: Total size of attached disks in GiB.
  • Nodes: Total number of nodes.
• Collects cluster availability information.
• Establishes a secure connection to the managing DP cluster.
• Sends metrics and cluster availability information to the managing DP cluster.
• Configures user authentication via the Dex provider of the managing cluster.
• If billing is enabled in the managing cluster, creates a PrometheusRemoteWrite custom resource to send metrics to the billing service of the managing DP cluster.

Module interactions

The module interacts with the following components:

1. Kube-apiserver: Manages resources of the managed cluster.

2. Prometheus: Retrieves metrics from the managed cluster.

3. Upmeter: Retrieves availability information of the managed cluster.

4. Dex: Synchronizes access tokens for Kubernetes API requests from the managing cluster to the managed cluster.

5. Managing cluster:

   • Organizes a proxy connection to the managing cluster.
   • Receives target cluster configuration.
   • Sends metrics to the billing service.

Additional resources

• Module commander documentation
• Module commander-agent documentation
• Module managed-postgresql documentation
• Module prometheus documentation
• Module terraform-manager documentation
• Module user-authn documentation