Kubernetes Ebook
Kubernetes Ebook
Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
Introduction to Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
Kubernetes Concepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
Kubernetes Components. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Kubernetes Workloads . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Useful Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2
Running GKE in Production. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
Lessons Learned. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3
Foreword
“Kubernetes is eating the container world.”
4
A lot of the documentation and resources online currently are
heavily geared toward developers. While this eBook leans toward
the technical side, our objective is to illustrate the benefits of
Kubernetes and provide enough context to equip your team to
with the information you need as you consider implementing
Kubernetes. Our eBook is by no means a comprehensive guide on all
things Kubernetes but rather a starting point to kick off discussion
around using it to deploy in the cloud and/or on the edge.
If you still have questions after reading this eBook, please don’t
hesitate to reach out. IoT may seem hard, but it doesn’t have to be.
— Team Leverege
5
1
Introduction to
Kubernetes
Introduction to Kubernetes
Introduction to Kubernetes
Kubernetes, or k8s for short, is an open-source container orchestrator.
Originally developed by the engineers at Google, Kubernetes
solves many problems involved with running a microservice
architecture in production. Kubernetes automatically takes care of
scaling, self-healing, load-balancing, rolling updates, and other
tasks that used to be done manually by DevOps engineers.
01 | Introduction 7
From Docker to Kubernetes
01 | Introduction 8
From Docker to Kubernetes
Docker Swarm
Docker is best-known for standardizing the container format which
solved the issue of managing multiple dependencies and binaries
across many application environments. With git-like semantics, Docker
images work intuitively and have come to influence the Cloud Native
Computing Foundation (CNCF) and Open Container Initiative (OCI).
As more and more of the community began to embrace Docker, the need
for a container orchestration solution naturally emerged. Before Docker
Swarm was introduced, Mesos and Kubernetes were actually recommended
by Docker as a container management solution in production.
Since then, Docker has become more than a mere Docker file
format provider. It has added Docker hub, registry, cloud, and other
services to become a true Platform-as-a-Service. One of the key
strengths of Docker Swarm is that if you’re already using Docker-
compose files, it’s really easy to get setup and start using it. However,
if you don’t want to depend on the entire Docker suite of tools,
then Docker Swarm isn’t as “open” as Kubernetes or Mesos.
01 | Introduction 9
From Docker to Kubernetes
Kubernetes
No other company can match Google’s extensive experience running
billions of containers. So when Google decided to donate Kubernetes—
based on their proprietary tool called Borg— to Cloud Native Computing
Foundation (CNCF) in 2014, Kubernetes quickly became the de facto
standard interface to abstract away the underlying infrastructure.
01 | Introduction 10
From Docker to Kubernetes
Mesos
Mesos is the oldest technology out of the three. It isn’t a true
container orchestrator but rather a cluster manager originally
designed to abstract away data center resources and scale
diverse workloads that may include stateless microservices
(containers), legacy monoliths, batch jobs, or data services.
01 | Introduction 12
Why Is Kubernetes Useful to Me?
Further Reading:
• Kubernetes vs. Mesos vs. Swarm - An Opinionated Discussion
01 | Introduction 13
2
Kubernetes Concepts
Kubernetes Concepts
Kubernetes Concepts
At a high level, Kubernetes works similarly to many cluster architectures.
It consists of one or more masters and multiple nodes that they control.
The master nodes orchestrate the applications running on nodes, and
they monitor them constantly to ensure that they match the desired
state the programmer has declared. In this chapter, we will dive deeper
into the key concepts underlying the Kubernetes Architecture. This
is by no means a comprehensive overview of every detail; for that,
you should read up on Kubernetes’s Official Documentation.
15
Kubernetes Components
Kubernetes Components
As mentioned previously, Kubernetes can largely be divided
into Master and Node Components. There are also some add-
ons such as the Web UI and DNS that are provided as a service
by managed Kubernetes offerings (e.g. GKE, AKS, EKS).
Master Components
Master components globally monitor the cluster and respond
to cluster events. These can include scheduling, scaling, or
restarting an unhealthy pod. Five components make up the
Master components: kube-apiserver, etcd, kube-scheduler,
kube-controller-manager, and cloud-controller-manager.
02 | Kubernetes Concepts 16
Kubernetes Components
Kubernetes Cluster
Node Components
Unlike Master components that usually run on a single node (unless High
Availability Setup is explicitly stated), Node components run on every node.
02 | Kubernetes Concepts 17
Kubernetes Object Management Model
02 | Kubernetes Concepts 18
Kubernetes Object Management Model
While this all sounds simple (and that was part of the intent),
it’s a powerful scheme that makes Kubernetes very useful. You
(the programmer) only have to specify the desired state, and
Kubernetes will take care of the rest (instead of having you run
specific commands to achieve this like in imperative models).
19
Kubernetes Components
02 | Kubernetes Concepts 20
Kubernetes Workloads
Kubernetes Workloads
Kubernetes workloads are divided into two major components:
pods (the basic building block) and controllers (e.g.
ReplicaSet, Deployment, StatefulSet, CronJob, etc.).
Pods
A Pod for Kubernetes is what a container is for Docker: the smallest
and simplest unit in its object model. It is helpful to conceptualize
Pods as a single instance of an application—or a container. In reality, a
Pod encapsulates one or more containers as well as storage resources,
an IP address, and rules on how the container(s) should run.
02 | Kubernetes Concepts 21
Kubernetes Workloads
Controllers
As mentioned earlier, Pods are usually deployed indirectly via
Controllers. The one used most frequently is Deployments, but
we will quickly cover some other types of Controllers.
ReplicaSet
ReplicaSet, as the name suggests, deploys the specified
replicas of the Pod. Unless you require custom updates to the
Pods, it’s recommended that you use Deployments, which
are higher level objects that wrap around ReplicaSets.
02 | Kubernetes Concepts 22
Kubernetes Workloads
Deployments
Deployments allow for rolling updates and easy rollbacks on top
of ReplicaSets. You can define the desired state in the Deployment
model, including scaling, rolling updates in canary or blue/
green fashion, and Deployments will take care of it for you.
StatefulSets
StatefulSets are similar to Deployments, but maintain a “sticky
identity” for each of the Pods. It’s useful for applications
in which a stable network identifier or persistent storage is
required. A common example would be ElasticSearch.
CronJobs
As the name suggests, CronJob manages time-based jobs.
Going with our ElasticSearch example, one common task would
be sending out daily reports or cleaning up old data.
02 | Kubernetes Concepts 23
3
Useful Tools
Useful Tools
Useful Tools
This chapter lists some useful Kubernetes tools we’ve used in production.
You can follow the official awesome-kubernetes for a comprehensive
list, but here is a short list of tools we’ve vetted and found useful.
Security
• Kubernetes Security Best Practice - Collection
of best practices for K8 security
Package Manager
• Helm - K8 package manager
Monitoring
• Prometheus-Kubernetes - Great sample Prometheus configs
03 | Useful Tools 25
Useful Tools
Testing
• Kube Monkey - Implementation of Netflix’s
Chaos Monkey for K8 clusters
CI/CD
• Keel - Lightweight CD that works well with Helm
Dev Tools
• kube-ps1 - K8 prompts for bash and zsh
Others
• Kubernetes Network Policy Recipes - Great
collection of sample network policy recipes
03 | Useful Tools 27
4
Monitoring
Monitoring
Monitoring
One of the downsides of microservice architecture is increased
complexity in monitoring. How does one monitor a cluster of distributed
applications that are communicating with each other? First, we
need to monitor the health of individual pods and applications. Is
the pod scheduled and deployed as intended? Are the applications
inside those pods running without errors and without performance
degradation? Second, we need to monitor the health of the entire
Kubernetes cluster. Is Kubernetes properly handling the resource
utilization of each node? What about the health of all of the nodes?
04 | Monitoring 29
Monitoring
In this chapter, we’ll examine some of the native tools Kubernetes provides
as well as some Google Cloud Platform (GCP)-specific and open-source
tools that we’ve found useful in production. Please note that this chapter
is by no means a comprehensive overview of all available monitoring
solutions for Kubernetes users. However, a combination of StackDriver
and Prometheus/Grafana have proved to be a robust and reliable
tool for our IoT deployments on Google Kubernetes Engine (GKE).
04 | Monitoring 30
Kubernetes Native Tools
04 | Monitoring 32
Stackdriver
Stackdriver
If you’re using the Google Kubernetes Engine, event exporter
for Stackdriver Monitoring is enabled by default if cloud
logging is enabled. For instructions on deploying to existing
clusters, please see the Official Documentation.
• Incidents
• Events
• CPU Usage
• Disk I/O
• Network Traffic
• Pods
04 | Monitoring 33
Stackdriver
04 | Monitoring 34
Prometheus + Grafana
Prometheus + Grafana
The following guide is heavily inspired by Sergey Nuzhdin’s post on
LWOLFS BLOG. He does a fantastic job of laying out how to deploy
Prometheus and Grafana to Kubernetes using Helm charts. However,
at the time of writing this, Nuzhdin does not cover deploying the
newer version of Prometheus (2.1) and configuring alerts. To fill
this knowledge gap, we created a Prometheus + Grafana Setup
Guide that goes through the setup process step-by-step.
Prometheus
Prometheus is an open-source, time-series monitoring tool
developed by the Cloud Native Computing Foundation (CNCF)
project—the foundation behind Kubernetes. It’s a flexible system
that can collect metrics, run complex queries, display graphs,
and trigger alerts based on custom rules. Default deployment of
Prometheus on Kubernetes scrapes all the aforementioned metrics
exposed by probes, cAdvisor, heapster, and kube-state-metrics.
04 | Monitoring 35
Prometheus + Grafana
Grafana
Grafana is an open-source software for time-series analysis and
visualization that has native plugins for Prometheus and other popular
libraries (Elasticsearch, InfluxDB, Graphite, etc.). While Prometheus
provides a rudimentary visualization functionality, it islimited to a simple
time-series graph format. Grafana allows for easier visualization of all of
the metrics exported by Prometheus to be consumed in various formats:
status checks, histograms, pie charts, trends, and complex graphs.
04 | Monitoring 36
5Deploying to
Cloud Providers
The State of Managed Kubernetes
Elastic Container Service (ECS) and Fargate, the community support and
involvement is neither as large nor as focused as it is with the other cloud
offerings. If you were already using ECS, the incentive to switch to Amazon
EKS may be small, unless you are exploring hybrid cloud solutions.
Lastly, AWS charges for master node usage. It charges $0.20/hr for
the master node plus usage for worker nodes for the cluster. Pricing is
always tricky to compare across cloud providers since billing is counted
slightly differently (not to mention heavily discounted enterprise deals),
but at face-value, Amazon EKS is significantly more expensive than
AKS or GKE since master node usage is not covered by the service.
Summary:
• Pros: Integration with other AWS tools, high number
of Availability Zones, and easy to migrate or integrate
if you’re already using AWS container options
• Cons: Expensive, steep learning curve and long setup time, and
relatively new so lacking certain Kubernetes-specific features
Despite an earlier release date than Amazon, AKS seems to lag behind
Amazon EKS and GKE in terms of Kubernetes upgrade, support, and
adoption numbers. While Amazon EKS and GKE supported Kubernetes
1.10.x relatively quickly since its release, AKS only recently started
supporting it. Additionally, unlike Amazon EKS and GKE, master nodes
are not offered in high-availability (HA) mode. Considering pre-existing
usage of Kubernetes on AWS and GKE’s early lead in Kubernetes
development, community involvement in AKS seems to be low so far.
That is not to say that AKS features are lagging in all domains. AKS
is currently the only provider for an incubator feature called Service
Catalog, which allows for applications inside Kubernetes to use externally
managed services (e.g. Azure databases). Microsoft has also been
active in maintaining a popular Kubernetes package manager, Helm.
Summary:
• Pros: Has the lowest cost, Kubernetes development is driven by
deep experience, and provides access to the newest features
Aside from lower cost, the main advantage of GKE will always be fast access
to the newest features and tools. GKE dashboard—along with Stackdriver
logging and monitoring agents already embedded on its VMs—allows for
easy monitoring of cluster health and usage. GKE also auto-scales your
nodes for you, which is a really nice feature that otherwise requires custom
setup when on Amazon EKS and AKS—this feature is great for quick
prototyping usage or load testing. Overall, GKE abstracts away a lot of the
infrastructure setup, simplifying the onboarding experience for new users.
the top, and Microsoft Azure, whose roots in the Windows ecosystem
enabled it to leverage deep ties to enterprise IT infrastructure.
Summary:
• Pros: Lowest cost, experienced team-leading Kubernetes
development, access to the newest feature
• Cons: GCP usage is lowest amongst the big three cloud providers
Best Price
First and foremost, GKE provides the best price for any managed
Kubernetes service. Unlike Amazon Elastic Container Service for Kubernetes
(Amazon EKS), which charges for the management of master nodes
($0.20/hr), GKE and Azure Kubernetes Service (AKS) only charge for the
virtual machines (VMs) running the K8 nodes. This means that master
node usage, cluster management (auto-scaling, network policy, auto-
upgrades), and other add-on services are provided free-of-charge. Due
to Google’s automatic sustained use discount and the option to use pre-
emptibles, we found that GKE ends up being slightly cheaper than AKS.
When you begin, you can easily spin up a cluster with a few clicks.
After enabling the Kubernetes Engine API, you can specify the VM
type,the number of nodes, and then click create. GKE takes care of
regional deployment, auto-scaling, auto-upgrades of both master
and child nodes, as well as network policy configuration via Calico.
While these are all things you can self-manage and deploy on AKS and
Amazon EKS, it’s a free service provided and managed by Google so
you can focus on application development on top of Kubernetes.
The best part of GKE, in our experience, is the cluster autoscaler feature.
GKE will automatically add or delete nodes when Kubernetes can no
longer schedules Pods on the existing nodes. Cluster autoscaler is tied to
a node pool, so you can autoscale different node pools when needed. This
feature really shines when you run a mixed node pool of pre-emptibles
and nodes reserved for intensive data workloads. Perhaps you elect
to run a node pool of pre-emptibles for non-critical workflow that can
scale on demand, while keeping some data-intensive or stateful sets on
another node pool. GKE will be Kubernetes-aware and scale automatically
for you, which is not provided by default on Amazon EKS or AKS.
Lessons Learned
After deploying some of the largest IoT systems in North America over the
past few years Leverege learned some lessons with Kubernetes the hard
way. Due to a huge community that Kubernetes has garnered, you can
now easily find lots of tips online. However, we recommend that you start
simple. Before messing around with complex namespace configurations,
Spinnaker/Istio, and pod policies, start with the default namespace
and learn the basics before you secure your cluster for production.
Manage Complexities
One of the downsides to Kubernetes, or perhaps microservice architectures
more generally, is how quickly complexities can grow. You’ll need
a good plan to manage complexities introduced by more layers or
monitoring of your cluster, nodes, pods, and applications will suffer.
Start out with consistent logging practices. Either pass context between
services or use a tracing system for better observability. Needless to
say, this will be extremely helpful when a cluster enters an unknown
state and you’ll be able to more easily track down the message flow.