The module lifecycle stage: Experimental

The module has requirements for installation

ClusterSecurityEventConfig with Loki (explicit allowlist)

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventConfig
metadata:
  name: default
spec:
  defaultSeverityThreshold: High
  enabledSources:
    - clusterSecurityEventShipper/kube-audit/kube-apiserver
  destinations:
    - cluster-loki

ClusterSecurityEventConfig with glob masks

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventConfig
metadata:
  name: default
spec:
  defaultSeverityThreshold: High
  enabledSourcesMasks:
    - clusterSecurityEventShipper/kube-audit/*
    - podSecurityEventShipper/*
  destinations:
    - cluster-loki

ClusterSecurityEventDestination (Loki)

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: cluster-loki
spec:
  type: Loki
  loki:
    endpoint: https://loki.d8-monitoring:3100
    auth:
      strategy: Bearer
      token: EXAMPLE
    tls:
      verifyCertificate: false
      verifyHostname: false

Credentials (token, password) can also be referenced from a Secret using tokenSecretRef/passwordSecretRef instead of inline values. The Secret must be in the d8-security-events-manager namespace with the key value in its data field and labeled security-events-manager.deckhouse.io/credential-secret: "true". Inline credential fields and their *SecretRef counterparts are mutually exclusive.

SecurityEventDefinition

apiVersion: security.deckhouse.io/v1alpha1
kind: SecurityEventDefinition
metadata:
  name: k8s-privilege-escalation
spec:
  code: K8S_PRIV_ESC
  category: Rbac
  severity: High
  description: "Attempt to create privileged pod or escalate permissions"
  source: kube-apiserver
  fields:
    - name: metadata.extra.privileges
      required: true

PodSecurityEventShipper (KubernetesPods with inline parser)

apiVersion: security.deckhouse.io/v1alpha1
kind: PodSecurityEventShipper
metadata:
  name: my-audit
  namespace: my-namespace
spec:
  - source: my-audit-app
    input:
      type: KubernetesPods
      kubernetesPods:
        labelSelector:
          matchLabels:
            app: audit
    parser:
      - name: app
        parser:
          type: Regex
          regex:
            patterns:
              - '^(?P<level>\w+)\s+(?P<msg>.+)$'
        fields:
          - name: level
            type: String
    produces:
      - eventCode: K8S_PRIV_ESC
        extract:
          field: message
          operator: Regex
          values:
            - '.*'
        transform:
          - key: event.severity
            value: level

ClusterSecurityEventShipper (File with parserRef)

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventShipper
metadata:
  name: kube-audit
spec:
  - source: kube-apiserver
    input:
      type: File
      files:
        - /var/log/kube-apiserver/audit.log
    parserRef: audit-json
    produces:
      - eventCode: K8S_AUDIT_FAIL

With a ClusterSecurityEventLoggingTransformationRules:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventLoggingTransformationRules
metadata:
  name: audit-json
spec:
  type: File
  file:
    paths:
      - /var/log/kube-apiserver/audit.log
    transform:
      parser:
        type: JSON
      fields:
        - name: stage
          type: String
        - name: responseStatus
          type: Int
      drop_raw: true

Multiple destinations (Loki + Splunk)

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: cluster-loki
spec:
  type: Loki
  loki:
    endpoint: https://loki.example:3100
---
apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: splunk-prod
spec:
  type: SplunkHEC
  splunkHEC:
    endpoint: https://splunk.example:8088
    token: YOUR_TOKEN
---
apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventConfig
metadata:
  name: default
spec:
  defaultSeverityThreshold: Medium
  enabledSourcesMasks:
    - "*"
  destinations:
    - cluster-loki
    - splunk-prod

SecurityEventLoggingTransformationRules (reusable parser)

apiVersion: security.deckhouse.io/v1alpha1
kind: SecurityEventLoggingTransformationRules
metadata:
  name: falco
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: runtime-audit-engine
  containers:
    - name: falco
      parser:
        type: JSON
      fields:
        - name: priority
          type: String
        - name: output
          type: String
      drop_raw: true

Enrichment via Plugin (k8s-pod-info)

A shipper can resolve a Pod field absent from the raw log by using enrich.source: Plugin. For the plugin reference (required input parameters, available output values, and the example enrich element template), see Enrichment plugins.

apiVersion: security.deckhouse.io/v1alpha1
kind: PodSecurityEventShipper
metadata:
  name: app-audit
  namespace: my-namespace
spec:
  - source: my-audit-app
    input:
      type: KubernetesPods
      kubernetesPods:
        labelSelector:
          matchLabels:
            app: audit
    parser:
      - name: app
        parser:
          type: JSON
    producesDefaults:
      # k8s.pod.name / k8s.namespace.name are populated by the KubernetesPods input.
      enrich:
        - target: actor.id
          source: Plugin
          plugin: k8s-pod-info
          value: serviceAccountName
          args:
            - key: pod_name
              value: k8s.pod.name
            - key: namespace
              value: k8s.namespace.name
    produces:
      - eventCode: K8S_PRIV_ESC
        extract:
          field: message
          operator: Regex
          values:
            - '.*'

Container-ID mode

When the raw event carries a container runtime ID instead of pod name/namespace (common in syscall/falco events), use the k8s-container-info plugin to resolve the serviceAccountName, name, or namespace in one lookup:

apiVersion: security.deckhouse.io/v1alpha1
kind: PodSecurityEventShipper
metadata:
  name: falco-audit
  namespace: kube-system
spec:
  - source: falco
    input:
      type: KubernetesPods
      kubernetesPods:
        labelSelector:
          matchLabels:
            app: falco
    parser:
      - name: falco
        parser:
          type: JSON
    producesDefaults:
      # container.id is populated by the parser from the falco JSON output.
      enrich:
        - target: actor.id
          source: Plugin
          plugin: k8s-container-info
          value: serviceAccountName
          args:
            - key: container_id
              value: container.id
    produces:
      - eventCode: K8S_PRIV_ESC
        extract:
          field: message
          operator: Regex
          values:
            - '.*'

Enrichment via Plugin (k8s-nodeuser-info)

A shipper can resolve a static-user username (nodeusers.deckhouse.io metadata.name) from the system UID carried in the event by using enrich.source: Plugin with the k8s-nodeuser-info plugin. For the plugin reference (required input parameters, available output values, and the example enrich element template), see Enrichment plugins.

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventShipper
metadata:
  name: falco-nodeuser-enrich
spec:
  - source: runtime-audit-engine
    input:
      type: KubernetesPods
      kubernetesPods:
        labelSelector:
          matchLabels:
            app: falco
    parser:
      - name: falco
        parser:
          type: JSON
    producesDefaults:
      # output_fields.user.uid is populated by the parser from the falco JSON output.
      enrich:
        - target: actor.name
          source: Plugin
          plugin: k8s-nodeuser-info
          value: username
          args:
            - key: uid
              value: output_fields.user.uid
    produces:
      - eventCode: K8S_SSH_LOGIN
        extract:
          field: message
          operator: Regex
          values:
            - '.*'

Enrichment via custom External plugin

Users can create External ClusterSecurityEventEnrichmentPlugin resources to register custom enrichment endpoints. Below is an example of an External plugin that resolves employee info by badge ID.

First, create the CSEP resource:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventEnrichmentPlugin
metadata:
  name: employee-info
spec:
  type: External
  description: "Resolve employee info by badge ID"
  endpoint:
    url: http://employee-enricher.hr-namespace.svc:8080/lookup
  args:
    - name: badge_id
      required: true
      description: "Event field path containing the employee badge ID"
  returns:
    fields:
      - name: username
        type: String
        description: "Employee username"
      - name: department
        type: String
        description: "Employee department"

Then, reference it in a shipper’s enrich rule:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventShipper
metadata:
  name: badge-enrich-example
spec:
  - source: runtime-audit-engine
    input:
      type: KubernetesPods
      kubernetesPods:
        labelSelector:
          matchLabels:
            app: falco
    parser:
      - name: falco
        parser:
          type: JSON
    producesDefaults:
      # output_fields.user.badge_id is populated by the parser.
      enrich:
        - target: actor.name
          source: Plugin
          plugin: employee-info
          value: username
          args:
            - key: badge_id
              value: output_fields.user.badge_id
    produces:
      - eventCode: K8S_BADGE_LOGIN
        extract:
          field: message
          operator: Regex
          values:
            - '.*'

Buffer configuration

Production setup: Disk + Block (default)

The default buffer settings are designed for production — Disk buffer with Block overflow behavior ensures zero data loss of security events during temporary destination outages.

Module config (ModuleConfig):

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: security-events-manager
spec:
  version: 1
  settings:
    gateway:
      buffer:
        type: Disk
        whenFull: Block
        maxSize: 512Mi
        maxEvents: 500
      logShipperBuffer:
        type: Disk
        whenFull: Block
        maxSize: 257Mi    # per node; must be > 256Mi, log-shipper's disk buffer minimum
        maxEvents: 500

Per-destination override in ClusterSecurityEventDestination, for example a larger buffer for a remote Elasticsearch:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: remote-elasticsearch
spec:
  type: Elasticsearch
  elasticsearch:
    endpoint: https://es.example.com:9200
    index: security-events
    auth:
      strategy: Bearer
      tokenSecretRef:
        name: es-token
  buffer:
    type: Disk
    whenFull: Block
    maxSize: 1Gi    # larger buffer for unreliable remote destination
    maxEvents: 500

Test setup: Memory + DropNewest

For test/dev environments where data loss is acceptable and you want maximum speed without backpressure:

Module config:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: security-events-manager
spec:
  version: 1
  settings:
    gateway:
      buffer:
        type: Memory
        whenFull: DropNewest
        maxSize: 50Mi
        maxEvents: 500
      logShipperBuffer:
        type: Memory
        whenFull: DropNewest
        maxSize: 50Mi
        maxEvents: 500

Alerting on security events

A ClusterSecurityEventAlertRule looks for a pattern in the collected security events and raises a Prometheus alert once the pattern occurs, for example “one actor read more than 20 secrets in 5 minutes” or “a subject was bound to cluster-admin”.

The pattern is expressed as a query in LogQL, the query language used to search Loki logs, and is evaluated against the cluster Loki on a schedule. Therefore, the cluster-loki destination must be enabled for any rule to work: securityEventConfig.destinations must include cluster-loki, as described in the module configuration reference. If the destination is missing or its type is not Loki, every alert rule is reported Degraded in status.conditions and no rule is evaluated.

A firing rule raises the D8SecurityEventAlertFiring alert, an ordinary Prometheus alert. It becomes a ClusterAlert object, appears in the console alongside other cluster alerts, and is delivered to every channel the cluster already sends alerts to. Delivery requires no configuration, and the alert disappears on its own 5–7 minutes after the rule stops matching. To list the created objects, run:

d8 k get clusteralerts

The pieces of a rule

Rules are placed in named groups. A group only organizes the rules and has no effect on how a rule is evaluated. Each rule in a group has the following fields:

  • alert — a short name for what it detects, for example MassSecretAccess.
  • mode — which of the two forms the rule is written in: Expr for a hand-written LogQL query, Match for the structured form. The field is required and must agree with the block that is set. It states explicitly what the presence of expr or match already implies, so that a reader of the manifest and the platform web interface both know the form of a rule without inspecting which block is filled.
  • expr or match — which events count towards this rule, and when it fires. Every rule has exactly one of the two, named by mode:
    • expr — the LogQL query that determines whether the rule fires. It must be a metric query, that is, one that returns a number, and must include the comparison itself, for example count_over_time({source_component="kube-audit"}[5m]) > 20. Nothing else defines the firing condition: reaching the threshold written into the query is what firing means. An example is given below.
    • match, with the optional aggregation, is the same condition built from structured fields: which event codes, sources, categories and actors are counted and, if aggregation is set, how many of them within which window trigger the alert. The form is compiled into the same kind of LogQL query that expr would hold, so it is a way of building that query rather than a separate evaluation path. See the section on the simple form below.
  • for is the time span over which expr must keep matching without a break before the rule fires. It filters out one-off spikes: with for: 5m, a spike that lasts a couple of minutes and then stops does not fire the rule. The default is 0s, meaning that the rule fires on the first match. The pending and firing wait is the same as for the for field of PrometheusRule.
  • annotations is the text the fired alert carries, and the field is required. The annotations.summary value must state what the rule detected. The full structure is given in the custom resource reference.
  • severityLevel and labels are described in the custom resource reference.

The structure mirrors PrometheusRule: named groups, each with an interval and a list of rules carrying alert, expr, for, labels and annotations, with LogQL used in expr instead of PromQL.

The simplest rule matches one event code and fires as soon as the event occurs:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: cluster-admin-bound
spec:
  groups:
    - name: default
      rules:
        - alert: ClusterAdminBound
          mode: Expr
          expr: |
            count_over_time({source_component="kube-audit", event_code="K8S_CLUSTER_ADMIN_BOUND"} [1m]) > 0
          severityLevel: 5
          annotations:
            summary: "A subject was bound to cluster-admin"

The query consists of the following parts:

  • the {...} selector picks the log streams to search;
  • count_over_time(...[1m]) counts the matching lines over the last minute;
  • > 0 is the condition itself.

Nothing more elaborate is required, because a cluster-admin binding is a rare event that deserves attention on every occurrence.

Both fields in the selector are stream labels, so Loki resolves them against its index and does not read the streams they exclude. Only the fields that stay in the event body require | json to become readable. Which fields belong to each group, and why a filter placed on the wrong side of that boundary is expensive, is described in the section on the specifics of writing rules below.

Counting events per actor

Most rules answer the question of whether an event happened too many times for one actor rather than whether it happened at all. Such rules use sum by (...), which keeps a separate count for each distinct value of a field instead of summing all the matching events together.

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: mass-secret-access
spec:
  groups:
    - name: default
      interval: 30s
      rules:
        - alert: MassSecretAccess
          mode: Expr
          # Platform actors read secrets continuously — on an idle cluster, one
          # DKP ServiceAccount produces thousands of reads an hour — so a rule
          # about people and workloads excludes the system actor pattern the
          # platform web interface's "Exclude system" filter also uses.
          expr: |
            sum by (actor_id) (
              count_over_time(
                {source_component="kube-audit",
                 event_code="K8S_SECRET_ACCESSED",
                 actor_id!="",
                 actor_id!~`system:serviceaccount:kube-system:.*|system:serviceaccount:d8-.*|system:node:.*|system:kube-.*|system:apiserver|kubernetes-admin`}
                [5m])
            ) > 20
          for: 0m
          severityLevel: 6
          annotations:
            summary: "One actor read an unusual number of secrets"
            description: "Actor {{ $labels.actor_id }} read secrets {{ $value }} times in 5m. Check whether the access was expected."

Compared to the first example, this rule adds the following:

  • a matcher excluding known platform identities (actor_id!~...), so that routine platform activity does not drown out the rule;
  • sum by (actor_id), which turns a single running total into a separate count per actor, so the rule fires once the count of any one actor crosses 20 rather than once the cluster-wide total does;
  • {{ $labels.actor_id }} and {{ $value }} in description, which quote the specific actor and count that triggered the alert. This is the same $labels and $value templating that Prometheus uses for the annotations of PrometheusRule.

Because the query grouped by actor_id, that field also becomes a label of the resulting alert. The section on the specifics of writing rules below explains why the list of such fields has to stay short.

The simple form

Writing LogQL by hand is not required. The match field selects events by structured fields, and the optional aggregation field counts them per group and fires once the count of a group reaches threshold within window. This is the same form that the module alert rules used before LogQL, compiled into a LogQL expr instead of driving a separate evaluation path. Without aggregation, a match rule fires on every matched event, the same way as the simplest rule shown above.

The following example rewrites the cluster-admin-bound rule using match:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: cluster-admin-bound
spec:
  groups:
    - name: default
      rules:
        - alert: ClusterAdminBound
          mode: Match
          match:
            eventCodes: ["K8S_CLUSTER_ADMIN_BOUND"]
          severityLevel: 5
          annotations:
            summary: "A subject was bound to cluster-admin"

The next example rewrites the mass-secret-access rule, using aggregation for grouping and the threshold. The excludeSystemActors parameter defaults to true, so it does not need to be specified explicitly:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: mass-secret-access
spec:
  groups:
    - name: default
      interval: 30s
      rules:
        - alert: MassSecretAccess
          mode: Match
          match:
            eventCodes: ["K8S_SECRET_ACCESSED"]
          aggregation:
            groupBy: ["ActorID"]
            threshold: 20
            window: "5m"
          for: 0m
          severityLevel: 6
          annotations:
            summary: "One actor read an unusual number of secrets"
            description: "Actor {{ $labels.actor_id }} read secrets {{ $value }} times in 5m. Check whether the access was expected."

The other fields of match — sources, categories, severityMin, actorType, actors and excludeActors — cover the same ground as the line filters of expr. Each of them is described in the custom resource reference. Use expr directly when a rule requires something that match cannot express: a field it does not cover, or a filter more specific than an exact or glob match, such as a numeric comparison on a JSON field.

Examples of rule configurations for different events

Example of a rule that fires an alert when one actor runs exec into several pods:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: repeated-pod-exec
spec:
  groups:
    - name: default
      rules:
        - alert: RepeatedPodExec
          mode: Expr
          expr: |
            sum by (actor_id) (
              count_over_time({source_component="kube-audit", event_code="K8S_POD_EXEC_ATTACH", actor_id!=""} [10m])
            ) > 5
          severityLevel: 6
          annotations:
            summary: "One actor opened shells in several pods"

Example of a rule that fires an alert on a burst of RBAC edits in one namespace (privilege escalation), grouped per actor–namespace pair:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: rbac-edit-burst
spec:
  groups:
    - name: default
      rules:
        - alert: RbacEditBurst
          mode: Expr
          expr: |
            sum by (actor_id, object_namespace) (
              count_over_time({source_component="kube-audit", event_code="K8S_RBAC_RESOURCES_MODIFIED"} | json [5m])
            ) > 5
          severityLevel: 5
          annotations:
            summary: "Several RBAC objects changed in a short window"
            description: "Compare {{ $labels.actor_id }} and {{ $labels.object_namespace }} against what was planned."

Example of a rule that fires an alert on brute force (repeated rejections from one address). Group by the client address rather than the actor, because an unauthorized request has no identity:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: unauthorized-burst
spec:
  groups:
    - name: default
      rules:
        - alert: UnauthorizedBurst
          mode: Expr
          expr: |
            sum by (actor_sourceIP) (
              count_over_time({source_component="kube-audit", event_code="K8S_UNAUTHORIZED_REQUEST"} | json [5m])
            ) > 10
          severityLevel: 5
          annotations:
            summary: "Repeated unauthorized requests from one address"

Example of a rule that fires on an event rare enough that a single occurrence deserves attention, deliberately at a severity that also blocks DKP updates:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: containerd-integrity
spec:
  groups:
    - name: default
      rules:
        - alert: ContainerdIntegrityViolation
          mode: Expr
          expr: |
            count_over_time({source_component="kube-audit", event_code="D8_CONTAINERD_INTEGRITY_VIOLATION"} [1m]) > 0
          severityLevel: 4     # Deliberately low: this event should also stop platform updates.
          annotations:
            summary: "containerd integrity check failed on a node"

Example of a rule that fires an alert when RBAC changes are initiated by subjects other than a ServiceAccount:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: person-touched-rbac
spec:
  groups:
    - name: default
      rules:
        - alert: PersonTouchedRbac
          mode: Expr
          expr: |
            count_over_time(
              {source_component="kube-audit",
               event_category="Rbac",
               actor_id!~`system:serviceaccount:.*`}
              [1m]
            ) > 0
          severityLevel: 5
          annotations:
            summary: "RBAC changed by a non-ServiceAccount identity"

A client certificate or an external identity is also not a ServiceAccount, so this is not quite “a human”. To watch a specific identity instead, match it directly:

          expr: |
            count_over_time(
              {source_component="kube-audit",
               event_code="K8S_SECRET_ACCESSED",
               actor_id=~`system:serviceaccount:my-app:.*|kubernetes-admin`}
              [1m]
            ) > 0

Specifics of writing rules

When writing rules, consider the following:

  • The rule applies only to events recorded by the cluster audit policy. The default DKP policy logs list of secrets but not a get of one secret by name, so a rule on K8S_SECRET_ACCESSED fires on listing secrets, not on reading a single known secret. The policy also drops most requests from the system:authenticated group, so human activity reaches the event stream only where an explicit rule records it: mutations in platform namespaces, RBAC changes, exec/attach, and similar. To confirm that the rule is not working, check the policy: an event that was never recorded cannot become an alert.
  • Platform actors are not excluded by default. Deckhouse’s own components generate most of the routine activity in the event stream — on an idle cluster, one platform ServiceAccount alone can produce thousands of secret reads an hour — so a rule about people and workloads needs a line filter excluding them, as the examples above do. The regex used there is the same one the console’s “Exclude system” filter uses, so the interface and your rules stay consistent: system:serviceaccount:kube-system:.*|system:serviceaccount:d8-.*|system:node:.*|system:kube-.*|system:apiserver|kubernetes-admin.
  • Filter on stream labels inside {...}, not after | json. The Loki destination publishes five fields as stream labels — source_component, event_code, event_severity, event_category and actor_id. A matcher on one of these placed inside the selector is resolved against Loki’s index, so the streams it excludes are never read at all. The same matcher written after | json makes Loki fetch every line in the window and parse it as JSON first, only to discard almost all of it — on every rule, on every interval. Every other field (object_namespace, object_type, actor_sourceIP, …) lives in the event body and does need | json, which is why two of the examples above still parse: they group by a body field. The match form picks the right side of this line on its own.
  • A cardinality limit protects Prometheus from a runaway rule. Grouping by a field with an unbounded set of values (an object name, a request path) can turn one rule into thousands of alerts. Past 500 distinct label sets for one rule, the rest are dropped — the rule keeps firing for the groups it already tracks, and it is reported CardinalityLimitExceeded in status.conditions so you notice.
  • {{ $value }} in an annotation multiplies Prometheus series. The rendered summary and description travel as labels of the firing metric, so each distinct rendering is a distinct series that then lives out the whole retention. A count that changes on every evaluation — the normal case for sum by (...) (count_over_time(...)) — therefore produces a fresh series per interval: at interval: 30s that is up to 120 series an hour, per group, on top of the groups themselves. Prefer {{ $labels.<name> }}, which stays stable for as long as the group does, and keep {{ $value }} for rules whose count moves rarely.
  • Evaluation state lives in memory, the same as most in-cluster components. Restarting the component that evaluates rules restarts every rule’s for countdown from scratch; a rule that was about to fire has to wait for again afterwards. The consequence to watch for is a for longer than the interval at which the component actually restarts: such a rule never reaches the end of its countdown and so never fires at all. Keep for well below that, or leave it at 0s and let the query’s own window do the smoothing.
  • severityLevel 4 or below blocks DKP updates on clusters where update.blockOnAlerts is enabled in the deckhouse module. For a critical event this can be a deliberate decision; for a routine one it is unexpected.
  • All rules raise an alert with the same name — D8SecurityEventAlertFiring. What tells one rule’s firing apart from another’s is a pair of labels: rule (the object name) and alert (the rule’s own alert: field). Search by those, not by the alert name. This differs from PrometheusRule, where each rule’s own alert: field becomes the Prometheus alert name directly — here that name is carried in a label instead, because every ClusterSecurityEventAlertRule rule feeds the one shared D8SecurityEventAlertFiring alert.

HTTP destination (webhook or HTTP collector)

Send security events with a POST request. Both framing modes exist because receivers interpret a batch of events differently: some read one JSON array per request, while most collectors read a stream of JSON records.

Example of sending security events from a DKP cluster to an external system over HTTP:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: siem-http
spec:
  type: Http
  http:
    endpoint: "https://collector.example.com/api/ingest"
    framing: NewlineDelimited      # One JSON object per line (NDJSON).
    auth:
      strategy: Bearer
      tokenSecretRef:
        name: siem-http-token
        key: token
    tls:
      verifyCertificate: true
    headers:
      X-Scope-OrgID: security
---
apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventConfig
metadata:
  name: default
spec:
  destinations:
    - siem-http
  enabledSourcesMasks:
    - "clusterSecurityEventShipper/*"
    - "podSecurityEventShipper/*"
  defaultSeverityThreshold: Low

Do not put an Authorization header in headers together with auth — the two conflict and the destination is rejected on admission.

Socket destination (CEF over syslog to SIEM)

Send security events in CEF format wrapped in RFC5424 syslog over TCP/TLS to a SIEM receiver:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: siem-syslog
spec:
  type: Socket
  socket:
    address: "siem.example.com:6514"
    mode: TCP
    tls:
      verifyCertificate: true
    encoding:
      codec: CEF
      syslogWrapper: RFC5424
      cef:
        deviceVendor: MyCompany
        deviceProduct: k8s-security
---
apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventConfig
metadata:
  name: default
spec:
  destinations:
    - siem-syslog
  enabledSourcesMasks:
    - "clusterSecurityEventShipper/*"
    - "podSecurityEventShipper/*"
  defaultSeverityThreshold: Low

Socket destination (raw JSON over UDP)

Send events as JSON over UDP (fire-and-forget, no TLS):

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: udp-collector
spec:
  type: Socket
  socket:
    address: "collector.example.com:514"
    mode: UDP

Socket destination (Unix socket for sidecar SIEM agent)

Send events to a local Unix domain socket, for example for a sidecar SIEM agent:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: sidecar-siem
spec:
  type: Socket
  socket:
    address: "/var/run/siem.sock"
    mode: Unix

Per-destination override:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: test-loki
spec:
  type: Loki
  loki:
    endpoint: http://loki:3100
  buffer:
    type: Memory
    whenFull: DropNewest
    maxSize: 50Mi
    maxEvents: 500