Auditing in Kubernetes: Security and Compliance for Yes, Even You!

Kubernetes has become the de facto standard for container orchestration in modern cloud-native environments. With the increasing adoption of Kubernetes, the need for auditing the activity within a Kubernetes cluster has become more important than ever before. Auditing in Kubernetes is the process of recording and analyzing the activity within a cluster. It provides valuable insight into the operations that are being performed, who is performing them, and when they are being performed.

What is Auditing in Kubernetes

At the heart of Kubernetes is the API server, which serves as the central hub for all API requests made to the cluster. As such, auditing at the API server is crucial to ensure the security and compliance of a Kubernetes cluster. In this article, we will explore what needs to be done on the API server to enable auditing in Kubernetes.

Auditing in Kubernetes is achieved by enabling the audit feature, which logs all API requests made to the Kubernetes API server. These logs can be used to investigate security incidents, troubleshoot issues, and comply with regulatory requirements. Kubernetes provides a flexible and extensible auditing framework that allows administrators to customize the audit policy and configure the audit backend.

Enabling Auditing in Kubernetes

The first step in enabling auditing in Kubernetes is to configure the API server to log all API requests. This can be achieved by setting the following flags in the kube-apiserver manifest file:

• — audit-log-path: This flag specifies the path to the audit log file. The audit log file stores all API requests made to the API server.
• — audit-log-maxage: This flag specifies the maximum age of the audit log file. Once the audit log file reaches this age, it is automatically rotated and a new audit log file is created.
• — audit-log-maxsize: This flag specifies the maximum size of the audit log file. Once the audit log file reaches this size, it is automatically rotated and a new audit log file is created.
• — audit-log-maxbackup: This flag specifies the maximum number of backup audit log files that can be retained.

Once these flags are set, the API server will start logging all API requests made to the cluster.

- --audit-policy-file=/etc/kubernetes/audit/policy.yaml
- --audit-log-path=/etc/kubernetes/audit/audit.log
- --audit-log-maxsize=500
- --audit-log-maxbackup=3

Audit log levels

The Kubernetes audit feature captures the following information for each API request:

• Request metadata: This includes the timestamp, HTTP method, requested URL, and the identity of the authenticated user.
• Request payload: This includes the request body and headers.
• Response metadata: This includes the HTTP status code, response headers, and response body.
• Annotations: This includes any metadata that was added to the request, such as labels or annotations.

The audit logs can be stored in various backend systems, such as log files, Syslog, or a remote storage service like Elasticsearch. Kubernetes supports multiple audit backend plugins, including file, syslog, webhook, and Elasticsearch. The audit logs can also be exported to third-party tools for analysis and visualization, such as Grafana or Kibana.

Configuring the audit policy

The audit policy in Kubernetes determines which API requests are logged and which are excluded. The audit policy is defined in a YAML file and can be applied to the API server using the kubectl apply command. The audit policy can be customized based on various criteria, such as user identity, the request method, resource type, and response status code.

Here is an example of an audit policy that logs all API requests made to the API server:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata

This policy logs only the metadata of each request, which includes the request and response headers, but not the request body.

Here is an example of an audit policy that logs only API requests made by a specific user:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
  userGroups:
  - my-user-group

This policy logs the request and response payloads of each API request made by a user belonging to the “my-user-group” group.

The audit policy can be customized based on the specific needs and requirements of the Kubernetes cluster.

This is how the audit logs look like

{
    "kind": "Event",
    "apiVersion": "audit.k8s.io/v1",
    "level": "RequestResponse",
    "auditID": "a6029022-4ff0-4c54-97ed-4099d0ca1923",
    "stage": "RequestReceived",
    "requestURI": "/api/v1/namespaces/default/serviceaccounts?fieldManager=kubectl-create",
    "verb": "create",
    "user": {
        "username": "kubernetes-admin",
        "groups": ["system:masters", "system:authenticated"]
    },
    "sourceIPs": ["172.31.22.88"],
    "userAgent": "kubectl/v1.23.6 (linux/amd64) kubernetes/ad33385",
    "objectRef": {
        "resource": "serviceaccounts",
        "namespace": "default",
        "apiVersion": "v1"
    },
    "requestReceivedTimestamp": "2022-07-31T08:36:48.679291Z",
    "stageTimestamp": "2022-07-31T08:36:48.679291Z"
}

Using the audit logs

Once the audit logs are generated, they can be analyzed and visualized using various tools, such as Elasticsearch, Grafana, and Kibana. The audit logs provide valuable insights into the activity within the cluster, allowing administrators to detect security incidents and compliance violations.

Best Practices for Kubernetes Auditing

In a production environment, you need to consider some best practices for Kubernetes Auditing. This includes:

• Create a comprehensive auditing policy based on your logging requirements
• Use webhook backend to send audit data to remote endpoints instead of storing logs on disk
• Control access to audit data so that it can’t be tampered with
• Configure alerts and visualizations based on the audit logs so that you get informed about important events, like the deletion of secrets etcetera

Conclusion

Auditing in Kubernetes is crucial to maintaining the security and compliance of a Kubernetes cluster. Enabling auditing at the API server and configuring the audit policy is a critical step in ensuring the audit logs capture the necessary information. By analyzing and visualizing the audit logs, administrators can gain valuable insights into the activity within the cluster and take the necessary steps to maintain its security and compliance.

Kubernetes Security Containers Auditing Docker