Kubernetes Secrets Handbook: Design, implement, and maintain production-grade Kubernetes Secrets management solutions

Kubernetes Secrets Handbook

Copyright © 2024 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Preet Ahuja

Publishing Product Manager: Suwarna Rajput

Senior Editor: Arun Nadar

Technical Editor: Irfa Ansari

Copy Editor: Safis Editing

Project Coordinator: Uma Devi

Proofreader: Safis Editing

Indexer: Tejal Daruwale Soni

Production Designer: Shankar Kalbhor

Marketing Coordinator: Rohan Dobhal

First published: January 2024

Production reference: 1120124

Published by

Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB

ISBN 978-1-80512-322-4

www.packtpub.com

To my father. A mentor for life and the best teacher I had. At every milestone reached, you have your own share of credit.

– Emmanouil Gkatziouras

To my grandmother for her kindness, my grandfather for his wisdom, and my partner and best friend, Mercedes Adams, for her love, patience, and continuous support.

– Rom Adams

To my wife. A beacon of love and strength in my life. Your support and care have shaped every success I’ve achieved. In every moment, your presence is a blessing beyond measure.

– Chen Xi

Foreword

In today’s digital landscape, the orchestration of containers has revolutionized how we build, deploy, manage, monitor, and scale cloud-native applications. Among the myriad tools available, Kubernetes has emerged as the de facto platform for container orchestration, empowering teams to streamline development and deployment processes like never before.

However, as we venture deeper into this realm of agility and efficiency, the critical aspect of security often becomes a concern relegated to the background. The management of Secrets – those sensitive pieces of information ranging from credentials, API keys, and other sensitive data – is a paramount challenge to organizations. Mismanagement of these Secrets can lead to substantial cyberattacks that jeopardize not just an organization’s data but also its reputation and trust. Even the accidental mismanagement of Secrets, such as Secrets being mistakenly stored in a code repository such as GitHub, can greatly increase the attack vector on both Kubernetes platforms and the applications that they host.

This book stands as a beacon in the sea of Kubernetes knowledge, guiding practitioners and enthusiasts alike through the intricate landscape of security and Secrets management within Kubernetes. It is a comprehensive guide that not only illuminates the potential vulnerabilities but also offers robust strategies and best practices to fortify your cloud-native applications and Kubernetes platforms.

With a meticulous approach, the authors delve into the core concepts of Kubernetes security, dissecting every layer of its architecture to unveil potential vulnerabilities and common pitfalls. Furthermore, they navigate the complex terrain of Secrets management, presenting battle-tested methodologies and tools to safeguard these invaluable assets.

From encryption in transit and encryption at rest to Secrets integration with CI/CD pipelines and mechanisms for identity and access management, this book thoroughly details the arsenal of security features Kubernetes offers, empowering you to craft and deliver a robust security strategy. It will arm you with practical insights and real-world examples, providing a hands-on approach to managing your Kubernetes Secrets against ever-evolving cyber threats.

As cloud-native application development continues its rapid evolution, the importance of securing our digital environments and artifacts cannot be overstated. This book is an indispensable companion, a guiding light for anyone navigating the Kubernetes ecosystem, ensuring that security and Secrets management remain at the forefront of their endeavors. It will cover Secrets management across multiple cloud providers and secure integration with other third-party vendors.

Prepare to embark on a journey that not only enhances your knowledge but also empowers you to fortify the foundation of your digital endeavors. When it comes to Kubernetes Secrets management, security should be built in, not bolt-on, and this book will arm you with the tools, techniques, and processes to ensure that your Secrets remain just that…secret!

Chris Jenkins, Principal Chief Architect, Global CTO Organization, Red Hat Inc.

Contributors

About the authors

Emmanouil Gkatziouras started his career in software as a Java developer. Since 2015, he has worked daily with cloud providers such as GCP, AWS, and Azure, and container orchestration tools such as Kubernetes. He has fulfilled many roles, either in lead positions or as an individual contributor. He enjoys being a versatile engineer and collaborating with development, platform, and architecture teams. He loves to give back to the developer community by contributing to open source projects and blogging on various software topics. He is committed to continuous learning and is a holder of certifications such as CKA, CCDAK, PSM, CKAD, and PSO. He is the author of A Developer’s Essential Guide to Docker Compose.

Rom Adams (né Romuald Vandepoel) is an open source and C-Suite advisor with 20 years of experience in the IT industry. He is a cloud-native expert who helps organizations to modernize and transform with open source solutions. He is advising companies and lawmakers on their open and inner-source strategies. He has previously worked as a principal architect at Ondat, a cloud-native storage company acquired by Akamai, where he designed products and hybrid cloud solutions. He has also held roles at Tyco, NetApp, and Red Hat, becoming a subject matter expert in hybrid cloud. He has been a moderator and speaker for several events, sharing his insights on culture, process, and technology adoption, as well as his passion for open innovation.

Chen Xi is a highly skilled Uber platform engineer. As a tech leader, he contributed to the secret and key management platform service, leading and delivering Secrets as a service with a 99.99% SLA for thousands of Uber container services across hybrid environments. His cloud infrastructure prowess is evident from his work on Google Kubernetes Engine (GKE) and the integration of Spire-based PKI systems. Prior to joining Uber, he worked at VMware, where he developed microservices for VMware’s Hybrid Kubernetes management platform (Tanzu Mission Control) and VMware Kubernetes Engine for multi-cloud (Cloud PKS). Chen is also a contributing author to the Certified Kubernetes Security Specialist (CKS) exam.

About the reviewers

Brad Blackard is an industry veteran with nearly 20 years of experience at companies such as Uber, Microsoft, and Boeing. At Uber, Brad led multiple technical initiatives as a leader in the Core Security organization, including Secrets management at scale. Most recently, Brad has served as head of engineering for DevZero, a start-up focused on securely improving developer experience and productivity, and he continues to serve there as an advisor.

Ethan Walton is a staff security engineer with a background in Kubernetes, DevOps, and cloud security. He has been active in the space since 2019, with work spanning platform engineering, cloud infrastructure consulting at Google, and leading cloud security initiatives within growing engineering organizations. Ethan is certified as a Google Cloud Professional Cloud Network Engineer and is an avid technology enthusiast. Outside of work, Ethan is also heavily invested in Venture Capital and helping to discover transformational technology start-up companies that will help shape the future.

I’d like to thank my family and especially my mother, father, and better half, Alexandra, for understanding the time and commitment it takes to continue pursuing my passion in the ever-changing world of technology. Day in and day out, this would not have been possible without them every step of the way. Thank you, and thanks to all the great technology trailblazers who continue to make every day an exciting day to work in this field.

James Skliros, a seasoned lead engineer, has shaped the digital landscape for over two decades, and he is renowned for spearheading projects and showcasing exceptional expertise in DevOps, the cloud, and Kubernetes. His adeptness at developing innovative initiatives and enhancing operational efficiency in DevOps is evident throughout his career. Evolving from a system administration background, he now focuses on architecture and solution design, emphasizing a passion for cloud security. Beyond his professional endeavors, he remains dedicated to technology, contributing insightful blogs and articles to his employer and personal platform.

I want to extend my deepest gratitude to my incredible wife, who has been my unwavering support during both the highs and lows of my career journey. Her steadfast encouragement has allowed me to persist in achieving my goals. Additionally, I appreciate Innablr for providing a growth-oriented workplace. Their support has played a key role in my career progression, and I am sincerely thankful for the opportunities they’ve offered.

Table of Contents

Preface

Part 1: Introduction to Kubernetes Secrets Management

1

Understanding Kubernetes Secrets Management

Technical requirements

Understanding Kubernetes’ origins and design principles

From bare metal to containers

Kubernetes overview

Kubernetes design principles

Kubernetes architecture

Getting hands-on – from a local container to a Kubernetes Pod

Secrets within Kubernetes

Secrets concepts

Storing Secrets on Kubernetes

Why should we care?

Security exposures

Summary

2

Walking through Kubernetes Secrets Management Concepts

Technical requirements

What are Kubernetes Secrets, and how do they differ from other Kubernetes objects?

Different types of Secrets and their usage scenarios

Opaque

Kubernetes service account token

Docker config

Basic authentication

TLS client or server

Token data

Conclusion

Creating, modifying, and deleting Secrets in Kubernetes

data and stringData

Updating Secrets

Deleting Secrets

Conclusion

Kubernetes Secrets configuration in different deployment scenarios

Secret usage among environments

From development to deployment

Conclusion

Requirement for managing Secrets, including secure storage and access control

Secure storage

Access control

Git and encryption

Conclusion

Securing access to Secrets with RBAC

RBAC introduction

RBAC and Secrets

Conclusion

Auditing and monitoring secret usage

minikube note

Summary

3

Encrypting Secrets the Kubernetes-Native Way

Technical requirements

Kubernetes-native encryption

Standalone native encryption

Native encryption with an external component

Going further with securing etcd

Linux system hardening

Linux data encryption

Transport

Summary

4

Debugging and Troubleshooting Kubernetes Secrets

Technical requirements

Discussion of common issues with Kubernetes Secrets

Helm and Helm Secrets

Secret application pitfalls

Debugging and troubleshooting Secrets

The describe command

Non-existing Secrets

Badly configured Secrets

Troubleshooting and observability solutions

Best practices for debugging and troubleshooting Secrets

Avoiding leaking Secrets

Summary

Part 2: Advanced Topics – Kubernetes Secrets in a Production Environment

5

Security, Auditing, and Compliance

Technical requirements

Cybersecurity versus cyber risk

Cybersecurity

Cyber risk

Conclusion

Compliance standards

Adopting a DevSecOps mindset

Tools

Trivy

kube-bench

Compliance Operator

StackRox

Kubernetes logging

Summary

6

Disaster Recovery and Backups

Technical requirements

Introduction to Secrets disaster recovery and backups

Importance of disaster recovery and backups for Secrets management

Practical case studies – the importance of backup Secrets

Backup strategies for Kubernetes Secrets

Geo-replication/cross-region replication

Point-in-time snapshots to immutable storage

Writing to multiple places during transit

Secrets versioning and backup considerations

Choosing a backup strategy

Security guidance for backup

Tools and solutions for backing up Kubernetes Secrets

Velero

etcdctl

HashiCorp Vault

AWS Secrets Manager

Azure Key Vault

Disaster recovery for Kubernetes Secrets

DRP in a Kubernetes environment

Regular testing and updating

Tools and solutions for disaster recovery in Kubernetes

Effective Secrets recovery scenario during a crisis

Summary

7

Challenges and Risks in Managing Secrets

Technical requirements

Grasping the complexities of Secrets management systems

General security risks in Secrets management

Secret zero

Secret access ballooning

Secret valet parking

Secret sprawl

Secret island

Challenges and risks in managing Secrets for Kubernetes

Security risks to manage Kubernetes Secrets

Mitigation strategies

Summary

Part 3: Kubernetes Secrets Providers

8

Exploring Cloud Secret Store on AWS

Technical requirements

Overview of AWS Secrets Manager

Encryption

Versioning

Rotation

Cloud-based features

Secrets Store CSI Driver

How Secrets Store CSI Driver works

Integrating AWS Secrets Manager with EKS

EKS cluster on AWS

Auditing

Kubernetes logs on CloudWatch

AWS Secrets Manager logs on AWS CloudTrail

KMS for AWS Secrets encryption

Provisioning KMS

Using KMS with EKS

Summary

9

Exploring Cloud Secret Store on Azure

Technical requirements

Overview of Azure Key Vault

Azure RBAC and access policy

High availability

Logging, auditing, and monitoring

Integration with other Azure components

Introduction to Workload Identity

Integrating an AKS cluster and Azure Key Vault

Configuring the Terraform project

Provisioning the network

Provisioning the AKS cluster

Creating a Key Vault

Auditing and logging

Azure Key Vault for secret encryption

Summary

10

Exploring Cloud Secret Store on GCP

Technical requirements

Overview of GCP Secret Manager

IAM

High availability

Logging, auditing, and monitoring

Integration with other Google Cloud components

Introduction to Workload Identity

Integrating GKE and GCP Secret Manager

Configuring the Terraform project

Provisioning the network

Provisioning a secret on Secret Manager

Provisioning the GKE cluster

Adding the CSI plugin for Kubernetes Secrets

Auditing and logging

GKE security posture dashboard

Integrating GKE and KMS

Summary

11

Exploring External Secret Stores

Technical requirements

Overview of external secret providers

Secrets Store CSI Driver

External secret store providers with CSI plugins

Secrets Injector

HashiCorp Vault

Using HashiCorp Vault as a secret storage

Vault and CSI Driver

Vault hosted on Kubernetes

Development mode versus production mode

CyberArk Conjur

How Conjur works

Qualities for securely managing Secrets

High availability

Encryption of data

Secure access

Versioning

Integration with Kubernetes

Auditing

Summary

12

Integrating with Secret Stores

Technical requirements

Configuring external secret stores in Kubernetes

Secret consumption in Kubernetes

Integrating with external secret stores

Kubernetes extensions and API mechanisms

Pod lifecycle and manipulation mechanisms

Specialized Kubernetes patterns – SealedSecrets

Secret Store CSI Driver for Kubernetes Secrets

Service mesh integration for secret distribution

Broker systems in Secrets management

Security implications and best practices

Practical and theoretical balance

Summary

13

Case Studies and Real-World Examples

Technical requirements

Real-world examples of how Kubernetes Secrets are used in production environments

Qualities of Secrets management in production

Secrets management from a CI/CD perspective

Integrating Secrets management into your CI/CD process

Risks to avoid with Secrets in CI/CD pipelines

Best practices for secure CI/CD Secrets management

Lessons learned from real-world deployments

Case study – Developing Secrets management

The Keywhiz Secrets management system at Square

Managing the Secrets lifecycle from end to end in a Kubernetes production cluster

Finalizing your decision on comprehensive Secrets lifecycle management

High SLAs as the key to business sustainability

Emergency recovery – backup and restore

Not just storing but provisioning Secrets

Secrets rotation

Authorization sprawl issue

Tagging, labeling, and masking on the client side

Auditing and monitoring on the server side

Ensuring secure Secrets distribution

Decommissioning and revoking Secrets

Responsibility, on-call support, penetration testing, and risk evaluation

Summary

14

Conclusion and the Future of Kubernetes Secrets Management

The current state of Kubernetes

Native solutions

External solutions

The future state of Kubernetes

Food for thought and enhancements

How to share your thoughts

Continuous improvement

Skill acquisition

Start early, fail fast, and iterate

Automation as a strategy and Everything as Code (EaC)

Threat modeling

Incident response

Summary

Index

Other Books You May Enjoy

Preface

Kubernetes Secrets management is a combination of practices and tools that help users to securely store and manage sensitive information, such as passwords, tokens, and certificates, within a Kubernetes cluster and keep them safe and secure. Securing Secrets such as passwords, API keys, and other sensitive information is critical for protecting applications and data from unauthorized access. Developers who understand Kubernetes Secrets management can help ensure that Secrets are managed securely and effectively, reducing the risk of security breaches. Many industries and regulatory frameworks have specific requirements for managing sensitive data. By learning Kubernetes Secrets management practices, developers can ensure that their applications comply with these requirements and avoid potential legal or financial penalties.

Who this book is for

This book is for software and DevOps engineers and system administrators looking to deploy and manage Secrets on Kubernetes. Specifically, it is aimed at the following:

Developers who are already familiar with Kubernetes and are looking to understand how to manage Secrets effectively. This could include individuals who are already using Kubernetes for application deployment, as well as those who are new to the platform and looking to learn more about its capabilities.

Security professionals who are interested in learning how to securely manage Secrets within a Kubernetes environment. This could include individuals who are responsible for securing applications, infrastructure, or networks, as well as those who are responsible for compliance and regulatory requirements.

Anyone who is interested in using Kubernetes to deploy and manage applications securely, and who wants to understand how to effectively manage Secrets within that environment.

What this book covers

Chapter 1

, Understanding Kubernetes Secrets Management, introduces you to Kubernetes and the importance of Secrets management in applications deployed on Kubernetes. It gives an overview of the challenges and risks associated with managing Secrets, the objectives, and the scope of the book.

Chapter 2

, Walking through Kubernetes Secrets Management Concepts, covers the basics of Kubernetes Secrets management, including the different types of Secrets; their usage scenarios; how to create, modify, and delete Secrets in Kubernetes; and secure storage and access control. It also covers how to securely access Secrets with RBAC and Pod Security Standards, as well as auditing and monitoring secret usage.

Chapter 3

, Encrypting Secrets the Kubernetes-Native Way, teaches you how to encrypt Secrets in transit and at rest in etcd, as well as key management and rotation in Kubernetes.

Chapter 4

, Debugging and Troubleshooting Kubernetes Secrets, provides guidance on identifying and addressing common issues that arise when managing Secrets in Kubernetes. It covers best practices for debugging and troubleshooting Secrets, including the usage of monitoring and logging tools, ensuring the security and reliability of Kubernetes-based applications.

Chapter 5

, Security, Auditing, and Compliance, focuses on the importance of compliance and security while managing Secrets in Kubernetes. It covers how to comply with security standards and regulations, mitigating security vulnerabilities, and ensuring secure Kubernetes Secrets management.

Chapter 6

, Disaster Recovery and Backups, provides you with an understanding of disaster recovery and backups for Kubernetes Secrets. It also covers backup strategies and disaster recovery plans.

Chapter 7

, Challenges and Risks in Managing Secrets, focuses on the challenges and risks associated with managing Secrets in hybrid and multi-cloud environments. It also covers strategies for mitigating security risks in Kubernetes Secrets management, guidelines for ensuring secure Kubernetes Secrets management, and the tools and technologies available for Kubernetes Secrets management.

Chapter 8

, Exploring Cloud Secret Store on AWS, introduces you to AWS Secrets Manager and KMS and how they can be integrated with Kubernetes. It also covers monitoring and logging operations on Kubernetes Secrets with AWS CloudWatch.

Chapter 9

, Exploring Cloud Secret Store on Azure, teaches you how to integrate Kubernetes with Azure Key Vault for secret storage, as well as the encryption of Secrets stored on etcd. It also covers monitoring and logging operations on Kubernetes Secrets through Azure’s observability tools.

Chapter 10

, Exploring Cloud Secret Store on GCP, introduces you to GCP Secret Manager and GCP KMS and how they can be integrated with Kubernetes. It also covers monitoring and logging operations on Kubernetes Secrets with GCP monitoring and logs.

Chapter 11

, Exploring External Secret Stores, explores different types of third-party external secret stores, such as HashiCorp Vault and CyberArk Secrets Manager. It teaches you how to use external secret stores to store sensitive data and the best practices for doing so. Additionally, the chapter also covers the security implications of using external secret stores and how they impact the overall security of a Kubernetes cluster.

Chapter 12

, Integrating with Secret Stores, teaches you how to integrate third-party Secrets management tools with Kubernetes. It covers external secret stores in Kubernetes and the different types of external secret stores that can be used. You will also gain an understanding of the security implications of using external secret stores and how to use them to store sensitive data using different approaches such as init containers, sidecars, CSI drivers, operators, and sealed Secrets. The chapter also covers the best practices for using external secret stores and how they can impact the overall security of a Kubernetes cluster.

Chapter 13

, Case Studies and Real-World Examples, covers real-world examples of how Kubernetes Secrets are used in production environments. It covers case studies of organizations that have implemented Secrets management in Kubernetes and lessons learned from real-world deployments. Additionally, you will learn about managing Secrets in CI/CD pipelines and integrating Secrets management into the CI/CD process. This chapter also covers Kubernetes tools to manage Secrets in pipelines and the best practices for secure CI/CD Secrets management.

Chapter 14

, Conclusion and the Future of Kubernetes Secrets Management, gives an overview of the current state of Kubernetes Secrets management and future trends and developments in the field. It also covers how to stay up to date with the latest trends and best practices in Kubernetes Secrets management.

To get the most out of this book

You should understand Bash scripting, containerization, and how Docker works. You should also understand Kubernetes and basic concepts of security. Knowledge of Terraform and cloud providers will also be beneficial.

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/Kubernetes-Secrets-Handbook

. If there’s an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/

. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: The kms provider plugin connects kube-apiserver with an external KMS to leverage an envelope encryption principle.

A block of code is set as follows:

apiVersion: apiserver.config.k8s.io/v1

kind: EncryptionConfiguration

resources:

  - resources:

      - secrets

    providers:

      - aesgcm:

          keys:

            - name: key-20230616

              secret: DlZbD9Vc9ADLjAxKBaWxoevlKdsMMIY68DxQZVabJM8=

      - identity: {}

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

apiVersion: v1

kind: ServiceAccount

metadata:

  annotations:

eks.amazonaws.com/role-arn

: arn:aws:iam::11111:role/eks-secret-reader

  name: service-token-reader

  namespace: default

Any command-line input or output is written as follows:

$ kubectl get events

...

11m         Normal    Pulled              pod/webpage                              Container image nginx:stable