Modeling Performance in Kubernetes
Modeling Performance in Kubernetes
net/publication/311622401
CITATIONS READS
99 5,904
4 authors, including:
Unai Arronategui
University of Zaragoza
27 PUBLICATIONS 307 CITATIONS
SEE PROFILE
All content following this page was uploaded by Unai Arronategui on 23 September 2019.
Permission to make digital or hard copies of all or part of this work for personal or 2. RELATED & BACKGROUND WORK
classroom use is granted without fee provided that copies are not made or distributed Virtual Machine virtualization and container virtualiza-
for profit or commercial advantage and that copies bear this notice and the full cita-
tion on the first page. Copyrights for components of this work owned by others than tion have attracted considerable research attention focusing
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re- on performance comparison using a suite of workloads that
publish, to post on servers or to redistribute to lists, requires prior specific permission stress CPU, memory, storage and networking resources [3,
and/or a fee. Request permissions from [email protected].
4, 5]. However, to the authors’ knowledge any work has
addressed container performance issues following a rigorous
c 2016 ACM. ISBN 978-1-4503-4616-0.
2
DOI: https://aws.amazon.com/es/lambda/details/
analytical approach. Unfortunately, most computer scien- The communication between Kubernetes master & slaves
tist are either not familiar with or reluctant to use formal (called minions in Kubernetes terminology) is realised through
methods. Even mature technologies, such as cloud comput- the kubelet service. This service must be executed on each
ing, provides a small portion of the work done on perfor- machine in the Kubernetes cluster. The node which acts as
mance considering formal models [6]. In [7, 8] is proposed master can also carry out the role of a slave during execution.
an iterative and cyclic approach, starting from the specifi- As Kubernetes works with Docker containers, the docker
cation of functional algorithms (specified in the functional daemon should be running on every machine in the cluster.
models). Then, it continues with the specification of the In addition, Kubernetes makes use of the etcd project to
computational resources available (in the operational mod- have a distributed storage system over all nodes, in order
els), providing complementary views: Control flow, data- to share configuration data. A master node runs an API
flow, and resources. The central role in this methodology server, implemented with a RESTful interface, which gives
is given to a set of Petri Net (PN) models describing the an entry point to the cluster. Kubernetes uses its API ser-
required functionality and the computational resources in- vice as a proxy to expose the services executing inside the
volved in the execution. PN is a well known formalism cluster to external applications/ users.
that combines simulation and analysis techniques. The for-
malism allow a developer to analyse the behaviour of the 2.1 Kubernetes Background
system throughout the development lifecycle and to gain The basic working unit in Kubernetes is a pod – an ab-
understanding of infrastructure and application behaviour. straction of a set of containers tightly coupled with some
In particular, PNs provide different analysis and prediction shared resources (the network interface and the storage sys-
techniques that allow developers to assess functional and tem). With this abstraction, Kubernetes adds persistence to
non-functional properties by means of simulation, and qual- the deployment of single containers. It is important to note
itative/quantitative analysis. Timed Petri net enriches the two aspect of a pod: (i) a pod is scheduled to execute on one
model with time, which enables the exploration of minimal machine, with all containers inside the pod being deployed
and maximal boundaries of performance and workload. As on the same machine; (ii) a pod has a local IP address inside
simulation tool, Petri nets allow the formulation of mod- the cluster network, and all containers inside the pod share
els with realistic features (as the competition for resources) the same port space. The main implication of this is that two
absent in other paradigms (as nude queueing networks). services which listen on the same port by default cannot be
Anyway, these models must be feed with temporal infor- deployed inside a pod. A pod can be replicated along several
mation, which have been the focus of previous works. Per- machine for scalability and fault tolerance purposes. When
formance evaluation has been done mainly for traditional a service or a set of services are deployed over several ma-
virtual machine execution in Clouds [9]. In [4, 10], Vir- chines, we can consider: (1) functional level or application
tual Machines and containers are compared attending sev- level involves exposing dependencies between the deployed
eral performance metrics. In the last few years, a few pro- services. Different services need to be coordinated in or-
posals have emerged to manage container clusters like Ku- der to provide a high level functionality. An example of this
bernetes and Docker Swarm 3 . Currently, Kubernetes seems kind of relationship is the deployment of a stream processing
to be the most featured and production grade. Limited re- infrastructure (e.g. Apache Kafka, Storm, Zookeeper and
search exists about container architecture & Kubernetes per- HDFS for persistence) or the GuestBook example provided
formance. In [5], a container performance study with Docker by Kubernetes, composed of a php frontend, a redis master
shows network performance degradation in some configu- and slave. Ubuntu Juju is a reference project which works at
rations and a negligible CPU performance impact in all the functional level to coordinate the deployment of services.
configurations. Network virtualization technologies (Linux (2) operational level or deployment level involves mapping
Bridge, OpenvSwitch) are pointed to as reasons, but mainly services to physical machines, VMs, pods or containers. It is
in full nested-container configurations where network vir- platform dependant and must involve isolation between re-
tualization is used twice. Unlike Docker, Kubernetes uses sources. Kubernetes primarily focuses on the operational/
a partial nested-container approach with the Pod concept deployment level. A pod implements a service, and coor-
where network virtualization is used once, as the same IP dination between different pods is achieved through global
address is used for all containers inside a Pod, leading to variables. Services running in others pods can be discovered
better performance. The Kubernetes team reports several through a DNS. This approach imposes some restrictions
performance metrics4 . They measured the response time of to Kubernetes. For instance, in the Guestbook example,
different API operations (e.g GET, PUT, POST operations Kubernetes’ scheduler cannot ensure that the three pods
over nodes and pods) and the Pod startup end-to-end re- are deployed rightly, because Kubernetes does not manage
sponse time. In their experiments, the 99th percentile pod the application level. The communication between pods is
startup time was below three seconds in a cluster with 1000 made at application level. Kubernetes makes use of a num-
nodes. Also, they propose Kubemark, a system to evaluate ber of support services, deployed in an isolated namespace
the performance of a Kubernetes cluster 5 . kube-system, such as a logging (fluentd) & monitoring ser-
Kubernetes is based on a master-slave architecture, with vice (heapster & Prometheus), a dashboard (Graphana), etc.
a particular emphasis on supporting a cluster of machines. Kubernetes also has a specific DNS server, deployed as an
add-on inside a pod.
3
https://www.docker.com/products/docker-swarm/
4
http://blog.kubernetes.io/2016/03/1000-nodes-and- 3. STATE & PERFORMANCE MODELS
beyond-updates-to-Kubernetes-performance-and-
scalability-in-12.html Kubernetes allows us to deploy several pods over physical
5 machines while it manages their lifecycle (i.e. start/stop
https://github.com/kubernetes/kubernetes/blob/release-
1.3/docs/proposals/kubemark.md new instances of running pods). We model a pod lifecycle
Figure 1: Model of Kubernetes manager system and shared resources
in order to estimate the impact of different scenarios on the Petri Nets [2] are a formal modelling tool that can be used
deployment time and the performance of the applications to describe and understand the behaviour of concurrent and
running inside the pod. According to the pod restart policy, distributed systems. A PN is a graph with two disjoint kinds
there are two kinds of pods in Kubernetes: (i) Service Pods of nodes: places and transitions. In high-level PNs, places
– expected to be in a permanent running state – which will represent a state of the system (or subsystem) and transi-
generate a background workload in the cluster, and they may tions represent the actions (or events) that change the state.
include Kubernetes system services (e.g. monitoring and Arcs going from a place to a transition represent precondi-
logging tools) or application services; (ii) Job/batch Pods – tions that need to be fulfilled in order to fire the transition.
expected to terminate on completion. The restart policy of Likewise, arcs from transitions to places represent postcon-
a job can be onFailure or never. the pod’s execution time is ditions fulfilled when a transition is fired.
dependant on the application deployed inside the containers. We use Object Nets [11] (a type of PN [2]) with reference
If the restart policy is onFailure, the time to deploy new semantics for our model, where a token net represents a task
pod instances in failure scenarios will have impact on the or agent, with a plan for the execution procedure, that can
total performance of the job or service. The performance be executed in different machines or locations. Token nets
metrics for both kinds of pods are different. For example, are called object nets in distinction to the system net to
for a service pod the deployment time is significant, along which it belongs. Interactions between objects nets, or be-
with the response time; while for a job pod the deployment tween the object nets and the system nets are represented
time and the total execution time (including restarting, if by labels as will be illustrated in the models. We have im-
necessary) are both useful metrics. Before a pod is executed, plemented our model in Renew [12] which uses Java as a
it requests resources from the Kubernetes scheduler such as language for the inscriptions and which allows us to use tu-
RAM and CPU. If there are enough resources available in ples as tokens. Additionally, in Renew, it is possible to pass
the cluster, the scheduler chooses the best node to deploy parameters to synchronisation inscriptions. The inscriptions
the pod. The CPU request is only taken into account when in the arcs match the tokens in the net using tuple notation.
CPU-intensive processes are running. When a container is For legibility concerns in the Nets figures, a double circle
idle (e.g. it is inside a service pod and the service has low place represent the place with the same name.
use at some point), other containers can utilise the unused In Kubernetes, the lifecycle of a pod depends on the state
CPU time. With this resource model, it is easy to see that (and consequently the lifecycle) of the containers that are
the total performance of the pod depends on its resource inside it. For instance, a pod has to wait until all its con-
requests and on the total workload at the node. tainers have been created. With the Object Nets abstrac-
tion, we can represent the Kubernetes manager system as
the System Net and the Pods (with the containers) as To-
3.1 Characterising Performance Models
Table 1: Timed transitions in the model
Transition Variable
T1 Time to create a container
T2 Execution time of a container
T3 Time until next failure in a container
T4, T5 Time to restart a container.
T6, T7 Graceful Termination of a container
50
30
20