Professional Documents
Culture Documents
Apache Hadoop Yarn Architecture PDF
Apache Hadoop Yarn Architecture PDF
1. Resource Manager: Runs on a master daemon and manages the resource allocation
in the cluster.
2. Node Manager: They run on the slave daemons and are responsible for the execution
of a task on every single Data Node.
3. Application Master: Manages the user job lifecycle and resource needs of individual
applications. It works along with the Node Manager and monitors the execution of
tasks.
4. Container: Package of resources including RAM, CPU, Network, HDD etc on a
single node.
Components of YARN
You can consider YARN as the brain of your Hadoop Ecosystem. The image below
represents the YARN Architecture.
Resource Manager
b) Application Manager
Node Manager
It takes care of individual nodes in a Hadoop cluster and manages user jobs and
workflow on the given node.
It registers with the Resource Manager and sends heartbeats with the health status of
the node.
Its primary goal is to manage application containers assigned to it by the resource
manager.
It keeps up-to-date with the Resource Manager.
Application Master requests the assigned container from the Node Manager by
sending it a Container Launch Context(CLC) which includes everything the
application needs in order to run. The Node Manager creates the requested container
process and starts it.
Monitors resource usage (memory, CPU) of individual containers.
Performs Log management.
It also kills the container as directed by the Resource Manager.
Application Master
An application is a single job submitted to the framework. Each such application has a
unique Application Master associated with it which is a framework specific entity.
It is the process that coordinates an application’s execution in the cluster and also
manages faults.
Its task is to negotiate resources from the Resource Manager and work with the Node
Manager to execute and monitor the component tasks.
It is responsible for negotiating appropriate resource containers from the
ResourceManager, tracking their status and monitoring progress.
Once started, it periodically sends heartbeats to the Resource Manager to affirm its
health and to update the record of its resource demands.
Container
It is a collection of physical resources such as RAM, CPU cores, and disks on a single
node.
YARN containers are managed by a container launch context which is container life-
cycle(CLC). This record contains a map of environment variables, dependencies
stored in a remotely accessible storage, security tokens, payload for Node Manager
services and the command necessary to create the process.
It grants rights to an application to use a specific amount of resources (memory, CPU
etc.) on a specific host.
Advantages
Better utilization of resources – The YARN framework does not have any fixed slots
for tasks. It provides a central resource manager which allows you to share multiple
applications through a common resource.
Running non-MapReduce applications – In YARN, the scheduling and resource
management capabilities are separated from the data processing component. This
allows Hadoop to run varied types of applications which do not conform to the
programming of the Hadoop framework. Hadoop clusters are now capable of running
independent interactive queries and performing better real-time analysis.
Backward compatibility – YARN comes as a backward-compatible framework, which
means any existing job of MapReduce can be executed in Hadoop 2.0.
JobTracker no longer exists – The two major roles of the JobTracker were resource
management and job scheduling. With the introduction of the YARN framework these
are now segregated into two separate components, namely:
o NodeManager
o ResourceManager