The managed-kafka module manages Apache Kafka instances in Deckhouse Platform (DP). Apache Kafka is a distributed Open Source data streaming platform and message broker. The module provides:

  • Automatic Deployment: Creates a Kafka instance using a simple YAML configuration.
  • Configuration Management: Separate KafkaClass resource for templating service creation and flexible validation of user parameters.
  • Long-term Storage: Supports PersistentVolumeClaim parameters for data.
  • Status: Displays the current status of the deployed Kafka instance.

For more details about module settings and usage examples, refer to the module documentation.

Module architecture

The following assumptions are used to simplify the diagram:

  • The diagram shows direct communication between containers in different Pods. In practice, components communicate through Kubernetes Services (internal load balancers). Service names are omitted when obvious from context. In other cases, a service name is shown above the arrow.
  • Pods can run with multiple replicas, but only one replica of each Pod is shown in the diagram.

The level-2 C4 architecture of the managed-kafka module and its interactions with other components of DP are shown in the following diagram:

Managed-kafka module architecture

Module components

The module consists of the following components:

  1. Managed-kafka-operator (Deployment): Kubernetes operator consisting of a single container manager that performs the following operations:

    • Reconciles Kafka custom resources in all user namespaces. The Kafka and KafkaClass custom resources define the settings of the Kafka instance.

    • Creates and manages StatefulSet, Service, Secret, and PersistentVolumeClaim resources related to the Kafka instance.

  2. Managed-kafka-webhook (Deployment): A component consisting of a single container manager.

    It validates and mutates Kafka and KafkaClass custom resources using the Validating/Mutating Admission Controllers mechanism.

  3. D8ms-kfk-<INSTANCE_NAME> (StatefulSet): A component consisting of a single kafka container that starts and runs the Kafka instance. It is created by the managed-kafka-operator component.

Module interactions

The module interacts with the kube-apiserver component:

  • Manages Kafka and Certificate custom resources.
  • Watches KafkaClass custom resources.
  • Manages StatefulSet, Service, Secret, and PersistentVolumeClaim resources.

The following external components interact with the module:

  1. Kube-apiserver: Validates and mutates Kafka and KafkaClass custom resources.

  2. User applications: Send requests to the Kafka instance.

Additional resources