Kubernetes API event audit

The Kubernetes auditing feature allows you to track requests to the API server and analyze events occurring in the cluster. Auditing can be useful for troubleshooting unexpected behavior and for meeting security requirements.

Kubernetes supports audit configuration via the Audit policy mechanism, which allows you to define logging rules for target operations. By default, audit results are written to the /var/log/kube-audit/audit.log file.

Built-in audit policies

Deckhouse Virtualization Platform (DVP) creates a default set of audit policies that log:

• Create, update, and delete operations on resources.
• Requests made on behalf of service accounts from the kube-system and d8-* system namespaces.
• Access to resources in the kube-system and d8-* system namespaces.

These policies are enabled by default. To disable them, set the basicAuditPolicyEnabled parameter to false.

Example:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: control-plane-manager
spec:
  version: 1
  settings:
    apiserver:
      auditPolicyEnabled: true
      basicAuditPolicyEnabled: false

Configuring a custom audit policy

To create an advanced Kubernetes API audit policy, follow these steps:

1. Enable the auditPolicyEnabled parameter in the control-plane-manager module configuration:

   apiVersion: deckhouse.io/v1alpha1
   kind: ModuleConfig
   metadata:
     name: control-plane-manager
   spec:
     version: 1
     settings:
       apiserver:
         auditPolicyEnabled: true
   

2. Create the kube-system/audit-policy Secret containing the policy YAML file encoded in Base64:

   apiVersion: v1
   kind: Secret
   metadata:
     name: audit-policy
     namespace: kube-system
   data:
     audit-policy.yaml: <Base64>
   

   Example audit-policy.yaml content with a minimal working configuration:

   apiVersion: audit.k8s.io/v1
   kind: Policy
   rules:
   - level: Metadata
     omitStages:
     - RequestReceived
   

   For more information on possible contents of audit-policy.yaml, refer to the following sources:

   • Kubernetes official documentation
   • GCE helper script code snippet

Working with the audit log file

On DVP master nodes, it is assumed that a log collection tool (log-shipper, promtail, or filebeat) is installed to monitor the /var/log/kube-audit/audit.log file.

The log rotation settings for this file are predefined and cannot be changed:

• Maximum file size: 1000 MB.
• Maximum retention period: 30 days.

Depending on the policy configuration and the volume of requests to the API server, the number of log entries can be very large. In such cases, the retention period may be reduced to less than 30 minutes.

If the API server fails to start, take the following steps:

1. Manually remove the --audit-log-* parameters from the /etc/kubernetes/manifests/kube-apiserver.yaml manifest.

2. Restart the API server with the following command:

   docker stop $(docker ps | grep kube-apiserver- | awk '{print $1}')
      
   # Alternative option (depending on the CRI in use).
   crictl stopp $(crictl pods --name=kube-apiserver -q)
   

3. After restarting, fix the Secret or delete it with the following command:

   d8 k -n kube-system delete secret audit-policy
   

Redirecting the audit log file to stdout

By default, the audit log is saved to the /var/log/kube-audit/audit.log file on master nodes. If necessary, you can redirect its output to the kube-apiserver process stdout instead of a file by setting the apiserver.auditLog.output parameter in the control-plane-manager module to Stdout:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: control-plane-manager
spec:
  version: 1
  settings:
    apiserver:
      auditPolicyEnabled: true
      auditLog:
        output: Stdout

In this case, the log will be available in the kube-apiserver container stdout.

Then, using the built-in DVP logging mechanism, you can configure log collection and forwarding to your own security system.