etcd in Kubernetes: Distributed Configuration Management

In the world of container orchestration, Kubernetes has emerged as the de facto standard for managing and scaling containerized applications. Behind the scenes, Kubernetes relies on a distributed key-value store called etcd to maintain crucial cluster state information and provide reliable configuration management. In this article, we will explore the role of etcd in Kubernetes and understand its significance in ensuring the resilience and consistency of the Kubernetes cluster.

What is etcd?

etcd is an open-source distributed key-value store that is designed to be highly available, consistent, and fault-tolerant. It is built on the Raft consensus algorithm, which ensures that data consistency is maintained across a cluster of nodes. etcd is written in Go and was developed by CoreOS, later becoming an integral part of the Kubernetes ecosystem.

Key Features of etcd

1. Distributed Data Store: etcd stores data as key-value pairs and replicates this data across multiple nodes in a Kubernetes cluster. This distribution ensures high availability and fault tolerance, as the cluster can still operate even if some nodes fail.
2. Strong Consistency: etcd ensures strong consistency using the Raft consensus algorithm. All nodes in the cluster agree on the state of the data, even in the presence of network partitions or node failures.
3. Watch Support: etcd provides a watch feature that allows applications to monitor changes to specific keys. This feature is crucial for implementing real-time event-driven workflows in Kubernetes.
4. Multi-Version Concurrency Control (MVCC): etcd uses MVCC to allow multiple clients to access and modify data simultaneously without interfering with each other. MVCC is essential for avoiding data conflicts in a distributed environment.

Role of etcd in Kubernetes:

In Kubernetes, etcd acts as the central brain of the cluster. It stores and manages all essential cluster state information, such as:

1. Configuration Data: Kubernetes stores configuration data, including Pod specifications, Service definitions, and ReplicaSet details, as key-value pairs in etcd.
2. Nodes and Components: etcd maintains information about all nodes in the cluster, along with the health status of various Kubernetes components like the kube-apiserver, kube-scheduler, and kube-controller-manager.
3. Service Discovery: Kubernetes services are registered in etcd, enabling dynamic service discovery within the cluster.
4. Secrets and ConfigMaps: Secrets and ConfigMaps, which hold sensitive data and configuration settings, respectively, are securely stored in etcd.

How etcd Ensures Reliability in Kubernetes:

1. High Availability: etcd achieves high availability by replicating data across multiple nodes in the cluster. In the event of a node failure, the data remains accessible from other healthy nodes, preventing service disruptions.
2. Consistency: With Raft consensus, etcd ensures that all nodes have the same view of the cluster state. This consistency guarantees that the Kubernetes components make decisions based on accurate and up-to-date information.
3. Automatic Data Recovery: etcd automatically recovers data from the remaining nodes if a node fails or rejoins the cluster. This capability ensures that the cluster remains operational with minimal downtime.
4. Backup and Restore: etcd allows administrators to back up the cluster state periodically, ensuring data durability and recoverability in case of catastrophic failures.

Running etcd Cluster

etcd cluster using Docker Compose is a convenient way to set up a local development environment for testing and experimentation. Here’s how you can create a simple etcd cluster using Docker Compose:

x-variables
	    flag_initial_cluster_token: &flag_initial_cluster_token '--initial-cluster-token=mys3cr3ttok3n'
	    common_settings: &common_settings
	        image: quay.io/coreos/etcd:v3.5.9
	        entrypoint: /usr/local/bin/etcd
	        ports:
	            - 2379
	

	services:
	    etcd-1:
	        <<: *common_settings
	        command:
	            - '--name=etcd-1'
	            - '--initial-advertise-peer-urls=http://etcd-1:2380'
	            - '--listen-peer-urls=http://0.0.0.0:2380'
	            - '--listen-client-urls=http://0.0.0.0:2379'
	            - '--advertise-client-urls=http://etcd-1:2379'
	            - '--heartbeat-interval=250'
	            - '--election-timeout=1250'
	            - '--initial-cluster=etcd-1=http://etcd-1:2380,etcd-2=http://etcd-2:2380,etcd-3=http://etcd-3:2380'
	            - '--initial-cluster-state=new'
	            - *flag_initial_cluster_token
	        volumes:
	            - etcd1:/etcd_data

	    etcd-2:
	        <<: *common_settings
	        command:
	            - '--name=etcd-2'
	            - '--initial-advertise-peer-urls=http://etcd-2:2380'
	            - '--listen-peer-urls=http://0.0.0.0:2380'
	            - '--listen-client-urls=http://0.0.0.0:2379'
	            - '--advertise-client-urls=http://etcd-2:2379'
	            - '--heartbeat-interval=250'
	            - '--election-timeout=1250'
	            - '--initial-cluster=etcd-1=http://etcd-1:2380,etcd-2=http://etcd-2:2380,etcd-3=http://etcd-3:2380'
	            - '--initial-cluster-state=new'
	            - *flag_initial_cluster_token
	        volumes:
	            - etcd2:/etcd_data
	
	    etcd-3:
	        <<: *common_settings
	        command:
	            - '--name=etcd-3'
	            - '--initial-advertise-peer-urls=http://etcd-3:2380'
	            - '--listen-peer-urls=http://0.0.0.0:2380'
	            - '--listen-client-urls=http://0.0.0.0:2379'
	            - '--advertise-client-urls=http://etcd-3:2379'
	            - '--heartbeat-interval=250'
	            - '--election-timeout=1250'
	            - '--initial-cluster=etcd-1=http://etcd-1:2380,etcd-2=http://etcd-2:2380,etcd-3=http://etcd-3:2380'
	            - '--initial-cluster-state=new'
	            - *flag_initial_cluster_token
	        volumes:
	            - etcd3:/etcd_data
	
	volumes:
	    etcd1:
	    etcd2:
	    etcd3::

start the etcd cluster using docker-compose

$ docker-compose -f docker-compose.yaml up -d

check if all the containers are in a healthy state

$ docker ps

CONTAINER ID   IMAGE                        COMMAND                  CREATED         STATUS         PORTS                                                   NAMES
3a4e906e81a6   quay.io/coreos/etcd:v3.5.9   "/usr/local/bin/etcd…"   8 minutes ago   Up 8 minutes   2380/tcp, 0.0.0.0:49164->2379/tcp, :::49164->2379/tcp   etcd-cluster-etcd-3-1
d3e82f47075b   quay.io/coreos/etcd:v3.5.9   "/usr/local/bin/etcd…"   8 minutes ago   Up 8 minutes   2380/tcp, 0.0.0.0:49165->2379/tcp, :::49165->2379/tcp   etcd-cluster-etcd-2-1
15a0857e99b5   quay.io/coreos/etcd:v3.5.9   "/usr/local/bin/etcd…"   8 minutes ago   Up 8 minutes   2380/tcp, 0.0.0.0:49166->2379/tcp, :::49166->2379/tcp   etcd-cluster-etcd-1-1

Then, you can run the etcdctl command to check the cluster health:

$ docker exec -it etcd-cluster-etcd-3-1 etcdctl member list

88d11e2649dad027, started, etcd-2, http://etcd-2:2380, http://etcd-2:2379, false
b8c6addf901e4e46, started, etcd-1, http://etcd-1:2380, http://etcd-1:2379, false
c3697a4fd7a20dcd, started, etcd-3, http://etcd-3:2380, http://etcd-3:2379, false

1. To write data to etcd, you use the put command:

 $ docker exec -it etcd-cluster-etcd-1-1 etcdctl put secret password

2. To read data from etcd, you use the get command:

$ docker exec -it etcd-cluster-etcd-2-1 etcdctl get secret
secret
password

3. To delete a key-value pair from etcd, you use the del command:

$ docker exec -it etcd-cluster-etcd-2-1 etcdctl del secret
1

Conclusion

In Kubernetes, etcd serves as a critical component responsible for maintaining the reliable state and configuration management of the entire cluster. Its distributed key-value store, combined with the Raft consensus algorithm, guarantees strong consistency, high availability, and fault tolerance. Thanks to etcd’s robustness, Kubernetes can efficiently manage containerized applications at scale, ensuring seamless orchestration and smooth user experiences.

As Kubernetes continues to evolve and grow in popularity, etcd will remain an indispensable pillar in the foundation of distributed systems, empowering administrators and developers to deploy and manage applications confidently in today’s dynamic and demanding cloud-native landscape.

Reference

1. https://www.ibm.com/topics/etcd
2. https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/
3. https://etcd.io/