managed-kafka module

Module Overview

The managed-kafka module manages Kafka clusters in Kubernetes clusters. It allows users to configure and scale Kafka clusters according to their needs, ensuring stable performance and predictable operations. Key features:

• Automatic deployment: Deploys a Kafka instance using simple YAML configuration.
• Configuration management: Separate KafkaClass resource for templating service creation and validating user configurations.
• Class-based governance: Supports sizing policies, overridable options, and CEL-based validation rules.
• Persistent Storage: Supports running with persistent volume claims for broker data.
• Status conditions: Exposes informative status conditions for tracking the deployed Kafka service.

Enabling the Module

To enable the managed-kafka module, apply the following manifest:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: managed-kafka
spec:
  enabled: true
  version: 1

After enabling the module, a cluster resource KafkaClass named default will be automatically created.

Quick Start

For module usage examples, see:

• section “Administrator Usage Examples”
• section “User Usage Examples”

Main Resources

The Managed Service Kafka module offers tools for two conditional roles User and Administrator. The main objects of the module are the Kafka, KafkaClass resources and the Kafka Operator state controller.

Conditional Administrator role:

• manages the cluster-wide KafkaClass resource and creates validation templates, deployment rules, and defaults for future services in DKP.
• has access to the system namespace d8-managed-kafka, where Kafka Operator is deployed, including its logs and operational state.

Conditional User role:

• manages the Kafka resource and configures the service according to constraints defined in KafkaClass by the Administrator.

Administrator Resources

KafkaClass

The KafkaClass resource allows restricting invalid configurations and predefining default values. Each Kafka resource must be associated with an existing KafkaClass resource. Before deploying the service, all user configuration is validated against the selected KafkaClass. Detailed resource description

Kafka Operator

When the module is enabled, the Kafka Operator controller is deployed in the system namespace d8-managed-kafka and reconciles the states of Kafka resources and resulting services in all user namespaces.

User Resources

Kafka

The main resource that describes the desired state of the service. It must be created inside a user namespace and serves as the source of truth for a specific deployed Kafka service. Detailed resource description