Tutorial-HDP-Administration V III

1 Big Data - Admin Course
Table of Contents
Table of Contents ...................................................................................................... 1

1. Prelude .................................................................................................................... 3
2. Ambari ..................................................................................................................... 5
3. Installing a HDP Cluster ..................................................................................... 19
4. Understanding YARN Config ............................................................................. 47
5. Map Reduce Job Submission – YARN .............................................................. 70
6. HDFS ACL ............................................................................................................. 83
7. Yarn – Two Nodes cluster ...................................................................................... 90
8. Using HDFS ........................................................................................................ 111
9. Change NN Heap settings – Services .............................................................. 146
10. Mounting HDFS NFS Gateway ....................................................................... 149
11. Using HDFS Snapshots ..................................................................................... 157
12. New – Configuring Cache .................................................................................. 168
13. Archival Disk Storage ........................................................................................ 180
14. Erasure Coding ................................................................................................... 187
15. Capacity scheduler ........................................................................................... 194
16. ROLLING BACK CONFIGURATION SETTING (Optional ) ..................... 214
1
17. NameNode HA ................................................................................................... 216

18. ResourceManager high availability ............................................................... 447
19. Tuning and Debugging HDP .......................................................................... 460
20. Managing YARN Config .................................................................................. 466
21. Miscelleanous ..................................................................................................... 478
Enable User Home Directory Creation. ..................................................................... 478
2
1. Prelude
All software will be in D:\\Software folder of your desktop.

All commands should be executed using putty.
Winscp browser should be used to copy software from the windows desktop to your linux
box.
Action: Start virtual machine.

You can start VM workstation or VM Player and import VM from d:/Software folder. After
that start the VM and connect to it using putty or directly from the console. Credentials to
connect to the system is root/life213. You can determine the ip of the system using
ifconfig command.
Refer the Supplement to import vmworkstation.
Note: Include the system ip and hostname in host file of the VM as well as in window client
machine so that you can access the system using hostname.
Ex : 10.10.20.21 tos.master.com
Mount the shared folder in VM host. Henceforth it will be refer as Software folder. You can
refer the supplement document for enabling and mounting shared folder.
Comment the last line in case of any issue as shown below.
Issue
3
Resolution (#vi /etc/fstab)
Reboot the machine to make the changes effect.
4
2. Ambari
Goal: You will install Ambari server on the host.
Hadoop requires java hence you need to install JDK and set Java Home.
#su - root
#mkdir /YARN
#tar -xvf jdk-8u181-linux-x64.tar.gz -C /YARN
mv jdk1.8.0_181 jdk
To include JAVA_HOME for all bash users, make an entry in /etc/profile.d as follows:
#echo "export JAVA_HOME=/YARN/jdk/" > /etc/profile.d/java.sh
Include in .bashrc (vi ~/.bashrc)

export PATH=$PATH:$JAVA_HOME/bin
5
We have extracted java in /YARN/jdk folder and specify java home using root logon.
(CentOS 7 64-bit-CLI)
Type bash in the command to prompt to reinitialized the scripts.
Next we will download the ambari repo so that yum utlility can download it.
Steps
1. #mkdir /apps
2. #cd /apps
3. Download the Ambari repository file to a directory on your installation host.
4. yum install wget
5.
wget -nv http://public-repo-
1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo -O
/etc/yum.repos.d/ambari.repo
Important
Do not modify the ambari.repo file name. This file is expected to be available on
the Ambari Server host during Agent registration.
6. Confirm that the repository is configured by checking the repo list.
yum repolist
6
1. Install the Ambari. This also installs the default PostgreSQL Ambari database.
yum install ambari-server

2. Enter y when prompted to confirm transaction and dependency checks
7
Ambari Server by default uses an embedded PostgreSQL database. When you install the
Ambari Server, the PostgreSQL packages and dependencies must be available for install.
These packages are typically available as part of your Operating System repositories.
Set Up the Ambari Server
Before starting the Ambari Server, you must set up the Ambari Server. Setup configures
Ambari to talk to the Ambari database, installs the JDK and allows you to customize the
user account the Ambari Server daemon will run as.
ambari-server setup
The command manages the setup process. Run the command on the Ambari server host to
start the setup process.
Respond to the setup prompt:
1. If you have not temporarily disabled SELinux, you may get a warning. Accept the
default (y), and continue.
2. By default, Ambari Server runs under root. Accept the default (n) at the Customize user
account for ambari-server daemon prompt, to proceed as root.
3. If you have not temporarily disabled iptables you may get a warning. Enter y to
continue.
4. Select a JDK version to download. Enter 1 to download Oracle JDK 1.8. Alternatively,
you can choose to enter a Custom JDK. If you choose Custom JDK, you must
manually install the JDK on all hosts and specify the Java Home path. :
8
For our lab Accept 2 and enter the Java home as - /YARN/jdk
Note
JDK support depends entirely on your choice of Stack versions. By default,
Ambari Server setup downloads and installs Oracle JDK 1.8 and the
accompanying Java Cryptography Extension (JCE) Policy Files.
5. Accept the Oracle JDK license when prompted. You must accept this license to
download the necessary JDK from Oracle. The JDK is installed during the deploy phase.
9
6. Select n at Enter advanced database configuration to use the default, embedded

PostgreSQL database for Ambari. The default PostgreSQL database name is ambari. The
default user name and password are ambari/bigdata. Otherwise, to use an existing
PostgreSQL, MySQL/MariaDB or Oracle database with Ambari, select y.
7. Setup completes.
Note
If your host accesses the Internet through a proxy server, you must configure
Ambari Server to use this proxy server.
10
Start the Ambari Server
 Run the following command on the Ambari Server host:
11
ambari-server start
 To check the Ambari Server processes:
ambari-server status
12
 To stop the Ambari Server: Do not execute this command. It’s for your information.
ambari-server stop
On Ambari Server start, Ambari runs a database consistency check looking for issues. If
any issues are found, Ambari Server start will abort and display the following
message: DB configs consistency check failed. Ambari writes more details about database
consistency check results to the/var/log/ambari-server/ambari-server-check-
database.log file.
You can force Ambari Server to start by skipping this check with the following option:
(optional when only there is issue)
ambari-server start --skip-database-check
If you have database issues, by choosing to skip this check, do not make any changes
to your cluster topology or perform a cluster upgrade until you correct the
database consistency issues.
If any error as shown below occurred during start up:

2019-04-13 20:25:48,248 INFO - Checking DB store version
13
2019-04-13 20:25:51,247 ERROR - Current database store version is not compatible

with current server version, serverVersion=2.7.3.0, schemaVersion=2.6.0
Solution:
# ambari-server status
Using python /usr/bin/python
Ambari-server status
Ambari Server not running. Stale PID File at: /var/run/ambari-server/ambari-server.pid
# ambari-server reset
Using python /usr/bin/python
Resetting ambari-server
**** WARNING **** You are about to reset and clear the Ambari Server database. This
will remove all cluster host and configuration information from the database. You will be
required to re-configure the Ambari server and re-run the cluster wizard.
Are you SURE you want to perform the reset [yes/no] (no)? yes
Confirm server reset [yes/no](no)? yes
Resetting the Server database...
Creating schema and user...
done.
Creating tables...
done.
Ambari Server 'reset' completed successfully.
Then configure the ambari set up again.
14
Next Steps
Log on to Apache Ambari
Prerequisites
Ambari Server must be running.
This will download required drivers for mysql
#yum install mysql-connector-java*
#cd /usr/lib/ambari-agent
#cp /usr/share/java/mysql-connector-java.jar .
#cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-agent/tmp/
Note: Whenever there is any issue related to jar file, determine the jar from the log file and
manually download and copy to the tmp folder as shown above,
15
log on to Ambari Web using a web browser and install the HDP cluster software.
Stop the firewall in the VM.
#systemctl stop firewalld
Steps
1. Point your web browser to
http://<your.ambari.server>:8080
,where <your.ambari.server> is the name of your ambari server host.
For example, a default Ambari server host is located
at http://tos.master.com:8080/#/login.
2. Log in to the Ambari Server using the default user name/password: admin/admin. For a
new cluster, the Cluster Install wizard displays a Welcome page.
16
Click Sign In
17
-------------------------------------------- Lab ends here ---------------------------------------------

-
18
3. Installing a HDP Cluster
Install MYSQL Server using root by opening a terminal.

su -
Download and add the repository, then update.
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
yum update
Install MySQL as usual and start the service. During installation, you will be asked if you
want to accept the results from the .rpm file’s GPG verification. If no error or mismatch
occurs, enter y.
yum install mysql-server
systemctl start mysqld
// Create User for Ranger KMS after upgrading mysql server
mysql_upgrade --force –uroot
19
Set the mysql root user id password to Life2134.

mysql
SET PASSWORD FOR 'root'@'localhost' = PASSWORD('Life2134');
FLUSH PRIVILEGES;
From the Ambari Welcome page, choose Launch Install Wizard.
20
Next Step
Name Your Cluster
Steps
1. In Name your cluster, type a name for the cluster you want to create.
Use no white spaces or special characters in the name.
Name it: myhdpcluster
2. Choose Next.
Choose the following Version
21
Select the HDP version to be installed i.e
22
Select the remote repository to download the required software:
23
2. Open a terminal and perform the following:

3. Generate public and private SSH keys on the Ambari Server host. (using root account)
ssh-keygen
4. Press Enter key twice to accept all default values.
5. Copy the SSH Public Key (id_rsa.pub) to the root account on your target hosts.
($HOME/.ssh/id_rsa.pub)
6. # cd $HOME/.ssh
.ssh/id_rsa
.ssh/id_rsa.pub
7. Add the SSH Public Key to the authorized_keys file on your target hosts.
cat id_rsa.pub >> authorized_keys
8. Depending on your version of SSH, you may need to set permissions on the .ssh
directory (to 700) and the authorized_keys file in that directory (to 600) on the target
hosts.
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
9. From the Ambari Server, make sure you can connect to each host in the cluster using
SSH, without having to enter a password.
ssh root@<remote.target.host>
where <remote.target.host> has the value of each host name in your cluster.
24
#ssh [email protected]
10. If the following warning message displays during your first connection: Are you sure
you want to continue connecting (yes/no)? Enter Yes.
11. Retain a copy of the SSH Private Key (id_rsa file) on the machine from which you will
run the web-based Ambari Install Wizard.
25
Above command will copy the private key to the /apps folder so that you can copy the file
on your laptop using the winscp tool.
26
Enter the hostname of the VM i.e hp.tos.com
27
Select id_rsa , you need to download the file in your window machine before selecting this
option.
28
It will take some times to highlight the Next button
29
Next
30
31
32
Accept all the options as shown above, some of the services are required by default.
33
34
Accept the default settings; ensure that NFS Gateway is selected.
35
All passwords should be Life2134 except the following.

Ranger password should be Life21345678
Select default setting after this screen and Go to Next steps.

Enter localhost for any server details if there are any errors.
36
Click on Ranger Tab
Execute the following by opening a terminal copying the my sql jar to appropriate folder:
#cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-server/resources
37
Enable the DB setup for ambari.

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-
java.jar
Set the entire password in this console to Life2134. – Ranger Tab
Verify the test connection and it should be Ok as shown above.

38
On the RANGER KMS tab,
Supply the db host as localhost and enter password as Life2134.
Click Next by accepting all default values till Deploy Button get displayed.
Reduce memory setting if any error occur due to insuffecient memory.
39
(Optional-)Reset the password setting as shown below in the customization advance

option. Life213
Click Deploy
40
Errors: In case of the following errors.. Copy the jar as mention below.
2019-10-15 03:29:04,579 - File['/usr/lib/ambari-agent/DBConnectionVerification.jar'] {'content':

DownloadSource('http://tos.hp.com:8080/resources/DBConnectionVerification.jar'), 'mode': 0644}
2019-10-15 03:29:04,606 - Not downloading the file from http://tos.hp.com:8080/resources/DBConnectionVerification.jar,

because /var/lib/ambari-agent/tmp/DBConnectionVerification.jar already exists
2019-10-15 03:29:04,638 - File['/var/lib/ambari-agent/tmp/mysql-connector-java.jar'] {'content':

DownloadSource('http://tos.hp.com:8080/resources/mysql-connector-java.jar')}
2019-10-15 03:29:04,639 - Downloading the file from http://tos.hp.com:8080/resources/mysql-connector-java.jar
2019-10-15 03:29:04,668 - The repository with version 3.1.4.0-315 for this command has been marked as resolved. It will be
used to report the version of the component which was installed
Command failed after 1 tries
#cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-server/resources
After copying the above jar, retry the installation.
41
In case the system doesn’t start for quite a long time i.e exceeds 10 minutes + verify the log
in the following location. View the latest file with out*txt.
/var/lib/ambari-agent/data/out*txt
If there is a message similar as shown below: Execute the next command to come out of
the safe mode.
42
2019-06-09 14:04:54,384 - Retrying after 10 seconds. Reason: Execution of

'/usr/hdp/current/hadoop-hdfs-namenode/bin/hdfs dfsadmin -fs
hdfs://tos.master.com:8020 -safemode get | grep 'Safe mode is OFF'' returned 1.
Verify the safemode:
# sudo -u hdfs hdfs dfsadmin -safemode get
#sudo -u hdfs hdfs dfsadmin -safemode leave
Task to stop or start any services.

Log on to Ambari serever click on any services listed below the services tab in the
dashboard. Click on the red icon and start from the menu option.
43
Hive Issues: (Optional) – unable to start Hive Metastore
Underlying cause: java.sql.SQLException : Access denied for user 'hive'@'tos.hp.com'

(using password: YES) SQL Error code: 1045
org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version.
At
org.apache.hadoop.hive.metastore.tools.HiveSchemaHelper.getConnectionToMetastore(
HiveSchemaHelper.ja
Solutions: Open a mysql console and grant the hive user access to the MySQL server.
Use the password which was supplied at the time of istalllation i.e is Life2134
# mysql -uroot –p
mysql> use mysql;
mysql> select user,host from user;
mysql> grant all on *.* to 'hive'@'tos.hp.com' identified by 'Life2134';
44
45
At the end of this step, there should be an entry of hive in the user table. Now you can start
the hive metastore and then the hive server. However you need to start the Ranger server
before starting the Hive server. Sequences to start the hive server are as follows.
 MySql Server
 Hive Metastore
 Ranger Admin , Ranger User Sync – Master and Slave.
 Hive Server
-------------------------------------------- Lab ends here ---------------------------------------------
46
4. Understanding YARN Config
Goal: You will verify some of the setting related to HDFS and YARN in the config file so
that you get familiarize with various config files.
ambari-server start
47
 To stop the Ambari Server: Do not execute this command. Its for your information.
ambari-server stop
You can view the log file in case of any issue: /var/log/ambari-server/ambari-server*.log
Log In to Apache Ambari
48
4. Log in to the Ambari Server using the default user name/password: admin/admin. You
can change these credentials later.
Action: Task to stop or start services.
49
Log on to Ambari server; click on any services listed below the services tab in the
http://tos.master.com:8080/#/main/dashboard/metrics
Start the hdfs and yarn services.
HDFS Sevices:
 NameNode
 DataNode
Yarn services:
 Resource Manager
 Node Manager
50
Namenode
In case the namenode take quite a long time to start up i.e exceeds 10 minutes+; verify the
log in the following location. View the latest file out of it.
If there is error message as shown below: Execute the command to come out of the safe
mode.

51
If everything goes well it will be as shown below:
52
Ensure that following services are started:
53
Open a terminal and vi file /etc/hadoop/conf/core-site.xml. verify the Port number and
the host that runs the namenode services.
<property>
<name>fs.defaultFS</name>
<value>hdfs://tos.hp.com:8020</value>
<final>true</final>
</property>
<property>
<name>hadoop.proxyuser.hdfs.groups</name>
<value>*</value>
</property>
You can also verify using the ambari console as shown below:
Ambari Dashboard  Services  HDFS  Configs  Advanced
54
This is the graphical representation of the config file. All changes have to be done from
the web console only so that synchronizing to all slaves node will be managed by ambari
else you have to do manually.
55
Verify the replication factor and the physical location of the data or block configured for
the cluster.
#vi /etc/hadoop/conf/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/apps/YARN/data/hadoop/hdfs/nn</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>file:/apps/YARN/data/hadoop/hdfs/snn</value>
</property>
<property>
<name>fs.checkpoint.edits.dir</name>
<value>file:/apps/YARN/data/hadoop/hdfs/snn</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/apps/YARN/data/hadoop/hdfs/dn</value>
56
</property>
<property>
<name>dfs.namenode.http-address</name>
<value>hp.tos.com:50070</value>
</property>
</configuration>
57
Verify /etc/hadoop/conf/mapred-site.xml
Map reduce related setting: as map reduce is executing in the yarn mode in this cluster.
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
yarn-site.xml, you can use vi editor to view The pluggable shuffle and pluggable sort
capabilities allow replacing the built in shuffle and sort logic with alternate
implementations.
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
58
Let us review Java Heap Sizes :
#vi /etc/hadoop/conf/hadoop-env.sh
Verify the following parameters.
HADOOP_HEAPSIZE="500"
HADOOP_NAMENODE_INIT_HEAPSIZE="500"
View the following information in mapred-env.sh

HADOOP_JOB_HISTORYSERVER_HEAPSIZE=250
Verify yarn-env.sh [If the following variable is not there in the file, you can make an entry
in the last line however ignore for the time being]
JAVA_HEAP_MAX=-Xmx500m
YARN_HEAPSIZE=500
59
After starting the HDFS, you can verify the java processes as shown below
#su -
#su hdfs
#jps
This command will list the java processes started for the Hadoop – Yarn.
#su -
#su hdfs
#jps
You can verify the Services using Web Interface also, You need to replace IP with that of
your server IP.
60
Access the Name Node UI and Data Node UI. Get familiarize the various features of these
UI especially the Node that belongs to the HDP cluster and the files store in the HDFS.
http://tos.master.com:50070/dfshealth.html#tab-overview
Verify the namenode that is use to connect to the cluster.
Hints : Overview 'tos.hp.com:8020' (active)
You can click on the various tabs to familiarize with the web UI.
61
It provides the overview of the Hadoop cluster.
62
You can verify the datanode information using this tab.

How many datanodes are there in the cluster? Now only one. What about any node being
decommision ?
63
Any snapshot being taken? You can verfy this after the snapshot lab. Here all snapshot
information will be stored.
64
You can verify the start up status of the hadoop cluster.
65
Browse the files in the cluster using the following option:
Click any of the file and verify it content.
66
You can access the Resource Manager UI as shown below:
http://tos.hp.com:8088/
Whenever you submit a job in the yarn cluster, you will get the job listed in this console.
How much resources consume will all be displayed here?
Verify the nodes that are running the yarn applications.
67
Understand the log folder and location: /var/log
Review the namenode log: tail -f hadoop-hdfs-namenode-tos.hp.com.log
68
Congrats! You have successfully completed Understanding main configuration of Yarn

Cluster.
-------------------------------------------- Lab ends here ---------------------------------------------

-
69
5. Map Reduce Job Submission – YARN
You will be able to submit map reduce job to hadoop yarn cluster at the end of this lab.
You need to ensure that Hadoop cluster is configured and started before proceeding ahead.
We are going to use sample MapReduce Examples provided by the hadoop installation
using hdfs user to understand how to submit MR job.
Run Sample MapReduce Examples using hdfs user

# su - hdfs
#export YARN_EXAMPLES=/usr/hdp/current/hadoop-mapreduce-client
#yarn jar $YARN_EXAMPLES/hadoop-mapreduce-examples-3.1.1.3.1.4.0-315.jar pi 16
1000
70
You can verify the execution of jobs using the YARN web console.
71
http://tos.hp.com:8088/ui2/#/cluster-overview
or To view the job status. Click on Resource Manager UI.
72
Then Click on Queues to understand which Queue executed the job.
73
We will talk about the Queues when we discuss scheduler later in the training.
Click on the Applications tab.
Now, the job is in accepted state. Finally it will be in running state as shown below.
74
75
Click on the application id link and verify the resources consume by this Job. Hover the
mouse over the color to get the exact value of memory consumption.
Find out where the AM executes for the job we have submitted now.
Click on Application ID using the RM UI.
76
In my case it’s the slave which can be different for your execution.
77
Click on Diagnostic to understand the resources demanded and consume.
In the above example its ask for 5 cotainers each of 768 mb and 1 v cores.
78
Finally at the end of the job execution, the pi result will be shown as above.
79
As shown above in the counter:

16 map tasks being launched for this job and only one reducer. Data locality happened for
all the 16 mappers.
80
Errata: /etc/hadoop/conf/yarn-site.xml
Issue Map reduces jobs not proceeding ahead and stuck at Accepted state.
Solution: verify the yarn-site.xml file and execute yarn services with yarn user only
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>124</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
</property>
Reduce the memory of Resource/Nodemanager to about 250 MB each if unable to execute

and start the history server of MR v2.
81
YARN  Configs  Edit
-------------------------------------------- Lab ends here ---------------------------------------------

-
82
6. HDFS ACL
HDFS ACLs give you the ability to specify fine-grained file permissions for specific named
users or named groups, not just the file’s owner and group.
To use ACLs, first you’ll need to enable ACLs on the NameNode by adding the following
configuration property to hdfs-site.xml and restarting the NameNode. Verify using ambari
UI.
<property>
<name>dfs.namenode.acls.enabled</name>
<value>true</value>
</property>
Services  HDFS  Configs  Advanced.
GRANTING ACCESS TO ANOTHER NAMED GROUP
83
This example demonstrates how a user ("hdfs"), shares folder access with colleagues from
another team ("hadoopdev"), so that the hadoopdev team can collaborate on the content of
that folder; this is accomplished by updating the default extended ACL of that directory:
Create a user henry in the user group – hadoopdev
Open a terminal and create an user – henry with password life213 using root logon
#su root
#groupadd hadoopdev
#useradd henry -G hadoopdev
#passwd henry
Create a folder /project using the hdfs user and dump a file.
#su hdfs
#hdfs dfs -mkdir /project
84
Create a file in the /app folder and update with the content below.
#vi /app/henry.txt
Content  “This file needs to be accesssed by the hadoopdev group also.”
Copy the above file to the hdfs /project folder using user hdfs else you will have permission
issues.
#hdfs dfs -copyFromLocal /apps/henry.txt /project
Open a new terminal and switch user to henry then display the file content.
#su henry
# hdfs dfs -cat /project/henry.txt
85
You will be able to display the content since the other group has – r_x i.e read access in
this directory.
Try copying file to the project using henry user id.
# hdfs dfs -copyFromLocal /apps/henry.txt /project/henry1.txt
Why is it deny? Verify the permission access of Other group. (hints – r_x)
Make the files and sub-directories created within the content directory readable and
writable by team "hadoopdev":
# su hdfs
$ hdfs dfs -setfacl -m group:hadoopdev:rwx /project
Set the default ACL setting for the parent directory:
86
$ hdfs dfs -setfacl -m default:group:hadoopdev:rwx /project
Create a sub-directory for the content you wish to share:
$ hdfs dfs -mkdir /project/dev
Inspect the new sub-directory ACLs to verify that HDFS has applied the new default
values:
$ hdfs dfs -getfacl -R /project
87
88
Note: At the time it is created, the default ACL is copied from the parent directory to the
child directory. Subsequent changes to the parent directory default ACL do not change the
ACLs of the existing child directories.
Now try copying a file using henry logon.

# hdfs dfs -copyFromLocal /apps/henry.txt /project/henry1.txt
It should be ok because we have assign write premisson using ACL to the project folder.
# hdfs dfs -cat /project/henry1.txt
------------------- ACL Lab Ends Here -----------------------------------------------------------
89
7. Yarn – Two Nodes cluster
From two single-node clusters to a multi-node cluster – We will build a multi-node cluster
using two Linux boxes in this lab.
Prerequisites
You have already installed HDP cluster on the master node i.e hp.tos.com using ambari.
Import the second machine in the VM workstation.
90
Henceforth, we will call the designated master machine just the master and the slave-only
machine the slave. We will also give the two machines these respective hostnames in their
networking setup, most notably in /etc/hosts. If the hostnames of your machines are
different (e.g. node01) then you must adapt the settings in this tutorial as appropriate.
Start both the VMs

For clarity let us rename the machine name as follows: Verify your IP and replace it
accordingly.
IP Alias hostname Remarks Services

10.10.20.20 Master (tos.master.com) First Machine As per the First Lab
10.10.20.21 Slave (tos.slave.com) New Machine NodeManager Data Node
On both the machines, rename it hostname by following the steps mention below, use root
credentials:
# nmtui.
91
92
You can verify the IPs as shown below:
93
Networking
Both the machines should be able to talk to each other over the network. You need to
enter the IPs of your respective machine in the hosts file of both the machines.
Update /etc/hosts on both machines with the following lines:
# vi /etc/hosts (for master AND slave)

10.72.47.42 master
10.72.47.27 slave
94
Enable SSH access so that both the machine can talk to each other without password.
The root user on the master must be able to connect to all the slave machines.
a) To its own user account on the master – i.e. ssh master in this context and not
necessarily ssh localhost – and
b) To the root user account on the slave via a password-less SSH login.
You have to add the root @master‘s public SSH key (which should be in
$HOME/.ssh/id_rsa.pub) to the authorized_keys file of root@slave (in this
user’s$HOME/.ssh/authorized_keys).
Perform the following steps for root on the master node
1. You need to configure Public/Private key for this, you can refer “Configuring SSH” in
the supplement to configure ssh on the master node.
2. You can do this manually or use the following SSH command on master node:
$ ssh-copy-id -i $HOME/.ssh/id_rsa.pub root@slave
95
This command will prompt you for the login password for user root on slave, then copy the
public SSH key for you, creating the correct directory and fixing the permissions as
necessary.
The final step is to test the SSH setup by connecting with user root from the master to the
user account root on the slave. The step is also needed to save slave‘s host key fingerprint
to the root@master‘sknown_hosts file.
3. So, connecting from master to master…
$ ssh master
…
96
4. And from master to slave.
root@master:~$ ssh slave
97
Hadoop YARN: Cluster Overview of Services. At the end you will have the following
services in the cluster of two nodes.
98
How the final multi-node cluster will look like.

The master node will run the “master” daemons for each layer:
 NameNode for the HDFS storage layer,
 and ResourceManage for the processing layer.
Both machines will run the “slave” daemons:
 DataNode for the HDFS layer,
 and NodeManager for processing layer.
Basically, the “master” daemons are responsible for coordination and management of the
“slave” daemons while the latter will do the actual data storage and data processing work.
Masters vs. Slaves
Typically one machine in the cluster is designated as the NameNode and another machine
the as ResourceManager, exclusively. These are the actual “master nodes”. The rest of
machines in the cluster act as both DataNode and NodeManager. These are the slaves or
“worker nodes”.
Ensure that java is installed and JAVA_HOME along with PATH variable are set as done
in the master.
Log in to Ambari (tos.master.com), click on Hosts and choose Add New Hosts from
the Actions menu.
99
100
Leave the other options as default.
Now I can click Register and Confirm.
It will take some times depending on the speed and resource.
In the Confirm Hosts step, Ambari server connects to the new node using SSH, it
registers the new node to the cluster and installs Ambari Agent in order to keep control
over it.Registering phase:
101
After sometimes it will be displayed as above If there is any issue click in the status link to
determine the root cause.
102
103
In step Assign Slaves and Clients; define my node to be a DataNode and has a
NodeManager installed as well (if you are running Apache Storm, Supervisor is also an
option).
Click next.
1. In step Configurations, there is not much to do, unless you operate with more than
one Configuration Group.
2. Click Next.
104
3. In step Review, one cans just doublecheck if everything is as planned.

Click deploy if everything is as it should be.
Installation will be started as shown below:
If there is any failure in the above steps verifies the log by clicking on the failed link and
performs the necessary steps. For example sql drivers updated manually in the master
node you needs to perform the same steps in the slave node too.
105
Next
After that addition of the slave node is complete.
106
You should have two nodes in the dashboard as shown below.
Background: The HDFS name table is stored on the NameNode’s (here: master) local
filesystem in the directory specified by dfs.name.dir. The name table is used by the
NameNode to store tracking and coordination information for the DataNodes.
Verify the Name node directory
#cd /hadoop/hdfs/namenode/current
#ls –ltr
107
Start the following services: Click on Hosts then click on each host listed on the menu to
start as below:
Master:
Name Node
Data Node
Resource Manager
Node Manager
Zookeeper
Slave:
DataNode
Node Manager.
http://10.10.20.20:50070/dfshealth.html#tab-datanode
108
You can verify the datanode using the Web UI – Name Node UI console:
109
You can view the node manager nodes with the url http://10.10.20.20:8088/cluster
Execute the MR Job on the two nodes cluster by refering “Map Reduce Job Submission –
YARN” section
-------------------------------------------- Lab ends here -----------------------------------------
110
8. Using HDFS
In this lab you will begin to get acquainted with the Hadoop tools. You will manipulate
files in HDFS, the Hadoop Distributed File System.
Set Up Your Environment
Before starting the labs, start up the VM and the HDFS, you need to logon with hdfs user
for this exercise.
ambari-server start
111
 To stop the Ambari Server: Do not execute this command. Its for your information.
112
ambari-server stop
You can view the log file in case of any issue: /var/log/ambari-server/ambari-server*.log
Log In to Apache Ambari
6. Log in to the Ambari Server using the default user name/password: admin/admin. You
can change these credentials later.
113
Action: Task to stop or start services.

Log on to Ambari server; click on any services listed below the services tab in the
http://tos.master.com:8080/#/main/dashboard/metrics
114
Start the hdfs and yarn services.
HDFS Sevices:
 NameNode
 DataNode
Yarn services:
 Resource Manager
 Node Manager
115
Namenode
In case the namenode take quite a long time to start up i.e exceeds 10 minutes+; verify the
log in the following location. View the latest file out of it.
If there is error message as shown below: Execute the command to exit the safe mode.

If everything goes well it should be as shown below:
116
Ensure that following services are started :
117
Data files (local), You need to copy all these files in your VM. All exercise need to be
performed using hdfs logon unless specified. You can create a data folder in your home
directory and dump all data inside that folder.
/software/data/shakespeare.tar.gz
/software/data/access_log.gz
/software/data/pg20417.txt
Hadoop is already installed, configured, and running on your virtual machine. Most of
your interaction with the system will be through a command-‐line wrapper called hadoop.
If you run this program with no arguments, it prints a help message. To try this, run the
following command in a terminal window:
# su - hdfs
$ hadoop
118
The hadoop command is subdivided into several subsystems. For example, there is a
subsystem for working with files in HDFS and another for launching and managing
MapReduce processing jobs.
Exploring HDFS
The subsystem associated with HDFS in the Hadoop wrapper program is called FsShell.
This subsystem can be invoked with the command hadoop fs.
Open a terminal window (if one is not already open) by double-‐clicking the Terminal icon
on the desktop.
In the terminal window, enter:
$ hadoop fs
119
You see a help message describing all the commands associated with the FsShell
subsystem.
Enter:
$ hadoop fs -ls /
This shows you the contents of the root directory in HDFS. There will be multiple entries,
one of which is /user. Individual users have a “home” directory under this directory,
120
named after their username; your username in this course is hdfs, therefore your home
directory is /user/hdfs.
Try viewing the contents of the /user directory by running:
$ hadoop fs -ls /user
You will see your home directory in the directory listing.
List the contents of your home directory by running:
$ hadoop fs -ls /user/hdfs
This is different from running hadoop fs -ls /foo, which refers to a directory that doesn’t
exist. In this case, an error message would be displayed.
121
Note that the directory structure in HDFS has nothing to do with the directory structure of
the local filesystem; they are completely separate namespaces.
Uploading Files
Besides browsing the existing filesystem, another important thing you can do with FsShell
is to upload new data into HDFS. Change directories to the local filesystem directory
containing the sample data we will be using in the homework labs.
$ cd /Software
If you perform a regular Linux ls command in this directory, you will see a few files,
including two named shakespeare.tar.gz and shakespeare-stream.tar.gz. Both of these
contain the complete works of Shakespeare in text format, but with different formats and
organizations. For now we will work with shakespeare.tar.gz.
Unzip shakespeare.tar.gz by running with root credentials(su root):
$ tar zxvf shakespeare.tar.gz
This creates a directory named shakespeare/ containing several files on your local
filesystem.
122
copy this directory into HDFS using hdfs:
$ hadoop fs -put shakespeare /user/hdfs/shakespeare
This copies the local shakespeare directory and its contents into a remote, HDFS directory
named /user/hdfs/shakespeare.
List the contents of your HDFS home directory now:
$ hadoop fs -ls /user/hdfs
You should see an entry for the shakespeare directory.
123
Now try the same fs -ls command but without a path argument:
$ hadoop fs -ls
You should see the same results. If you don’t pass a directory name to the –ls command, it
assumes you mean your home directory, i.e. /user/hdfs.
Relative paths
If you pass any relative (non-absolute) paths to FsShell commands (or use relative paths
in MapReduce programs), they are considered relative to your home directory.
124
We will also need a sample web server log file, which we will put into HDFS for use in
future labs. This file is currently compressed using GZip. Rather than extract the file to the
local disk and then upload it, we will extract and upload in one step.
First, create a directory in HDFS in which to store it:
$ hadoop fs -mkdir weblog
Now, extract and upload the file in one step. The -c option to gunzip uncompressed to
standard output, and the dash (-) in the hadoop fs -put command takes whatever is being
sent to its standard input and places that data in HDFS.
$ gunzip -c access_log.gz | hadoop fs -put - weblog/access_log
Run the hadoop fs -ls command to verify that the log file is in your HDFS home directory.
The access log file is quite large – around 500 MB. Create a smaller version of this file,
consisting only of its first 5000 lines, and store the smaller version in HDFS. You can use
the smaller version for testing in subsequent labs.
$ hadoop fs -mkdir testlog
$ gunzip -c access_log.gz | head -n 5000 | hadoop fs -put - testlog/test_access_log
Viewing and Manipulating Files

Now let’s view some of the data you just copied into HDFS.
125
Enter:
$ hadoop fs -ls shakespeare
This lists the contents of the /user/hdfs/shakespeare HDFS directory, which consists of
the files comedies, glossary, histories, poems, and tragedies.
The glossary file included in the compressed file you began with is not strictly a work of
Shakespeare, so let’s remove it:
$ hadoop fs -rm shakespeare/glossary
Note that you could leave this file in place if you so wished. If you did, then it would be
included in subsequent computations across the works of Shakespeare, and would skew
your results slightly. As with many real-‐world big data problems, you make trade-‐offs
between the labor to purify your input data and the precision of your results.
126
Enter:
$ hadoop fs -cat shakespeare/histories | tail -n 50
This prints the last 50 lines of Henry IV, Part 1 to your terminal. This command is handy
for viewing the output of MapReduce programs. Very often, an individual output file of a
MapReduce program is very large, making it inconvenient to view the entire file in the
terminal. For this reason, it’s often a good idea to pipe the output of the fs -cat command
into head, tail, more, or less.
127
To download a file to work with on the local filesystem use the fs -get command. This
command takes two arguments: an HDFS path and a local path. It copies the HDFS
contents into the local filesystem:
$ hadoop fs -get shakespeare/poems ~/shakepoems.txt

$ less ~/shakepoems.txt
There are several other operations available with the hadoop fs command to perform most
common filesystem manipulations: mv, cp, mkdir, etc.
128
$ hadoop fs
This displays a brief usage report of the commands available within FsShell. Try playing
around with a few of these commands if you like.
Basic Hadoop Filesystem commands (Optional)
In order to work with HDFS you need to use the hadoop fs command. For example to list
the / and /app directories you need to input the following commands:
hadoop fs -ls /
hadoop fs -ls /tmp
There are many commands you can run within the Hadoop filesystem. For example to
make the directory test you can issue the following command:
hadoop fs -mkdir test
Now let's see the directory we've created:

hadoop fs -ls /
hadoop fs -ls /user/hdfs
You should be aware that you can pipe (using the | character) any HDFS command to be
used with the Linux shell. For example, you can easily use grep with HDFS by doing the
following:
hadoop fs -mkdir /user/hdfs/test2

129
hadoop fs -ls /user/hdfs | grep test
As you can see the grep command only returned the lines which had test in them (thus
removing the "Found x items" line and oozie-root directory from the listing.
In order to move files between your regular linux filesystem and HDFS you will likely use
the put and get commands. First, move a single file to the hadoop filesystem.
Copy pg20417.txt from software folder to data folder
hadoop fs -put /home/hdfs/data/pg20417.txt pg20417.txt

hadoop fs -ls /user/hdfs
130
You should now see a new file called /user/hdfs/pg* listed. In order to view the contents of
this file we will use the -cat command as follows:
hadoop fs -cat pg20417.txt
We can also use the linux diff command to see if the file we put on HDFS is actually the
same as the original on the local filesystem. You can do this as follows:
diff <( hadoop fs -cat pg20417.txt) /home/hdfs/data/pg20417.txt
131
Since the diff command produces no output we know that the files are the same (the diff
command prints all the lines in the files that differ).
Some more Hadoop Filesystem commands
In order to use HDFS commands recursively generally you add an "r" to the HDFS
command (In the Linux shell this is generally done with the "-R" argument) For example,
to do a recursive listing we'll use the -lsr command rather than just -ls. Try this:
hadoop fs -ls /user

hadoop fs -ls -R /user
132
In order to find the size of files you need to use the -du or -dus commands. Keep in mind
that these commands return the file size in bytes. To find the size of the pg20417.txt file
use the following command:
hadoop fs -du pg20417.txt
To find the size of all files individually in the /user/root directory use the following
command:
hadoop fs -du /user/hdfs
To find the size of all files in total of the /user/root directory use the following command:
hadoop fs -dus /user/hdfs
133
If you would like to get more information about a given command, invoke -help as
follows:
hadoop fs -help
For example, to get help on the dus command you'd do the following:
hadoop fs -help dus
You can observe the HDFS’s namenode console as follows:
Familiarize the various options
http://10.10.20.20:50070/dfshealth.html#tab-overview
134
135
Click on Datanodes
136
Click on Snapshot
137
Click on Startup Progressand utlities
138
Click on Logs
http://192.168.246.131:50070/logs/
You can verify the log by clicking on the datanode log file.
139
You can verifying the Hadoop File System Health. Check for minimally replicated blocks if
any.
#hadoop fsck /
140
#hadoop dfsadmin –report
This is a dfsadmin command for reporting on each DataNode. It displays the status of
Hadoop cluster. Any under replicated blocks or Corrupt replicas?
141
#hadoop dfsadmin -metasave hadoop.txt

This will save some of NameNode’s metadata into its log directory under filename.
In this metadata, you’ll find lists of blocks waiting for replication, blocks being replicated,
and blocks awaiting deletion. For replication each block will also have a list of DataNodes
being replicated to. Finally, the metasave file will also have summary statistics on each
DataNode.
Go to log folder:
# cd /var/hadoop/logs
# ls
# vi hadoop.txt
142
You can get the information of hadoop safe mode.

#hadoop dfsadmin -safemode get
hadoop dfsadmin -safemode enter
hadoop dfsadmin -safemode leave
143
hadoop fsck /
144
You can determine the version of Hadoop.
#hadoop version
The default port is 50070. To get a list of files in a directory you would use:
1. curl -i "http://tos.hp.com:50070/webhdfs/v1/user/root/output/?op=LISTSTATUS"
-------------------------------------------- Lab ends here ---------------------------------------------
145
9. Change NN Heap settings – Services
Set the Namenode java heap size (Memory) to 2.5 GB using the following option
Use Services > [HDFS] > Configs to optimize service performance for the service.
1. In Ambari Web, click a service name in the service summary list on the left.
2. From the the service Summary page, click the Configs tab, then use one of the
following tabs to manage configuration settings.
o Use the Configs tab to manage configuration versions and groups.
o Use the Settings tab to manage Smart Configs by adjusting the green, slider buttons.
o Use the Advanced tab to edit specific configuration properties and values.
146
147
3. Click Save.
4. Enter a description for this configuration version that includes your current changes.
5. Review and confirm each recommended change.
Restart all affected services.
-------------------------------------------- Lab ends here ---------------------------------------------

-
148
10. Mounting HDFS NFS Gateway
Objective: To learn how to mount HDFS to a local file system.
Successful Outcome: HDSF will be mounted to /home/hdfs/hdfs--‐mount.

NFS utills will be installed in mater
NFS Gatway service will be started in slave
NFS access will be done in master.
Before You Begin: SSH into node1 i.e master.
Step 1: Install NFS – On Master

1.1. Execute following yum command to install NFS on master:
# yum -y install nfs*
Step 2: Configure NFS

2.1. Within Ambari, stop the HDFS service. (Services  HDFS  ActionS 
Stop [Confirm stop])
149
2.2. Go to Services - ‐> HDFS - ‐> Configs, scroll down to the

Advanced section and click on the Advanced heading to expand
the section of General.
2.3. Locate the dfs.namenode.accesstime.precision (Access time
precision) property and set it to 3600000.
2.4. Scroll down to the "Custom hdfs--‐site.xml" section.

Using the "Add Property..." link, add the following new property:
dfs.nfs3.dump.dir=/tmp/.hdfs-nfs
150
2.5. Click the Save button to save your changes.
2.6. Start the HDFS service back up.
Ensure that the following value is set appropriately
151
Extra Hints – Optional:

<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
<description>
Set this to '*' to allow the gateway user to proxy any group.
</description>
</property>
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
<description>
Set this to '*' to allow requests from any hosts to be proxied.
</description>
</property>
Step 3: Start NFS on Slave using root logon id.
3.1. Run the following commands to stop the nfs and rpcbind services.
(If they are not running, the following commands will fail, which is
no problem):
152
3.2. Now start the NFS services using the hadoop script –Open new terminal for each of
the command in slave using root user id:
3.3. Verify that the required services are up and running.from the master console.
# rpcinfo -p tos.slave.com
You should se output similar to the following:
153
3.4. Verify that the HDFS namespace is exported and can be mounted
by any client from the master console.
# showmount -e tos.slave.com
You should see output similar to the following:
Step 4: Mount HDFS to the Local File System on master

4.1. As the root user, create a new directory on your local file system named
154
#mkdir –p /home/hdfs/hdfs-mount.
4.2. Change its ownership to the hdfs user.
[mkdir -p /home/hdfs/hdfs-mount
chown hdfs /home/hdfs/hdfs-mount]
4.3. Execute the following command on a single line to mount HDFS to
the hdfs- mount directory:
4.4. Browse HDFS now on your local file system.
# ls -l /home/hdfs/hdfs-mount
155
You can display the content of the file from the local OS.
-------------------------------------------- Lab ends here -------------------------------------------
156
11. Using HDFS Snapshots
Objective: Understand how snapshots work in Hadoop.
Successful Outcome: The data folder has a snapshot taken of it.
Before You Begin: SSH into node1 i.e master aka tos.master.com
Step 1: Load a File into HDFS

1.1. Start by creating a new directory in HDFS:
# su - hdfs
# hadoop fs -mkdir data
1.2. Put a file in the new directory:
# hadoop fs -put constitution.txt data/
1.3. Run the fsck command on the file:
157
Select--‐and--‐copy the block ID.
158
1.4. Use the find command to identify the location of the block on your
local file system. The command will look like the following, and you may
need to run it on master or slave:
159
Use root user id to find the block.

# find / -name "blk_1073743693_2875*"
1.5. Which node and folder is the block stored in?

You can also verify using the Namenode UI.
Name Node UI  Utilities  Browse File System  User  Hdfs  Data 
constitution.txt
160
As seen above the block is replicated to hp and slave since we have a default replication
factor of 3 and we have only 2 nodes in the cluster.
Step 2: Enable Snapshots
2.1. Now let’s enable the /user/hdfs/data directory for taking snapshots:
161
You should see the following confirmation:
Step 3: Create a Snapshot

3.1. Now create a snapshot of /user/hdfs/data:
# hdfs dfs -createSnapshot /user/hdfs/data ss01
You should see the following confirmation:
3.2. Verify the snapshot was created by viewing the contents of the
data/.snapshot folder:
162
3.3. Try to delete the data folder:
# hadoop fs -rm -R data
You cannot delete the folder.
Step 4: Delete the File

4.1. Delete the constitution.txt file in data:
# hadoop fs -rm data/constitution.txt
4.2. Use the ls command to verify the file is no longer in the data folder in HDFS.
163
# hadoop fs -ls -R data/
4.3. Check whether the file still exists in

/user/hdfs/data/.snapshot/ss01. It should still be there.
4.4.
Step 5: Recover the File

5.1. Let’s copy this file from data/.snapshot/ss01 to the data directory.
# hadoop fs -cp data/.snapshot/ss01/constitution.txt data/
5.2. Run the fsck command again on data/constitution.txt. Notice

that the block and location information have changed for this file.
164
#hdfs fsck /user/hdfs/data/constitution.txt -files -

blocks –locations
Using Name Node UI
165
Earlier Block ID was ending with 4 now its 5. Block ID changes since its iss a new block.
You can view the Block snapshot using the web ui too – Name Node UI..
166
-------------------------------------------- Lab ends here --------------------------------------------
167
12. New – Configuring Cache
In this lab we will configure the cache directive and store two files in the cache pool.
Then we will execute some commands to determine the stats of the pools and the cache
being used.
Ensure that the following folder and file exists before configuring the cache. We are
going to cache the files in this folder. If the following files are not present then refer hdfs
lab to store it.
#hadoop fs -ls /user/hdfs/weblog
If it doesn’t exist upload the files using the following command:

# hadoop fs -copyFromLocal excite-small.log weblog/
We are going to cache these particular files.
168
Let us find out any cache directives and Pools configure in the cluster.
#hdfs cacheadmin –listPools
# hdfs cacheadmin –listDirectives
0 results mean nothing is configured as of now.

Create a cache pool with the identifier “cPTos”
#hdfs cacheadmin -addPool cPTos
You can get the stats of all the pools that are configure in the cluster or just a single pool
by specifying the pool name along with the command.
# hdfs cacheadmin -listPools -stats
# hdfs cacheadmin -listPools -stats cPTos
169
Let us add the file excite-small.log in the cache by executing the following command.
#hdfs cacheadmin -addDirective -path /user/hdfs/weblog/excite-small.log -pool cPTos
Let us get the stats of the pool after adding the file.
#hdfs cacheadmin -listPools -stats cPTos
170
Here you can get the stats of the pool cPTos.

Verify the cache directives now:
#hdfs cacheadmin -listDirectives
# hdfs cacheadmin -addDirective -path /user/hdfs/weblog/access_log -pool cPTos

After adding another file you can verify the directives again.
Let us update the capacity of the Pool size.
# hdfs cacheadmin -modifyPool cPTos -limit 400000
171
DFS Report Before configuring Cache:
In the above report cache configured and use are still 0.

172
Although we have configured Caching, there are certains steps to be completed before it
become effective.
Let us remove the directive using the following commands. You can determine the
directive ids using the list directive command used before.
#hdfs cacheadmin -removeDirective 1
#hdfs cacheadmin -removeDirective 2
You need to configure some OS related setting before the caching become effective.
Change the OS Setting i.e memory limit.
. Set memlock limits on each datanode. This will take effect after you logout and login again. Use root login for these
comands
1. # On each datanode (max cacheable memory in KB) example for 1.0 GB
echo "* hard memlock 604541532 " >> /etc/security/limits.conf
echo "* soft memlock 604541532 " >> /etc/security/limits.conf
173
Modify the pool to increase the capacity of the pool:

#hdfs cacheadmin -modifyPool cPTos -limit 604541532 -mode 0777
Change HDFS configurations to support Caching. Add the following to HDFS configs, Custom hdfs-site.xml in Ambari.
This example is for 0.5 GB (in bytes)
1. dfs.datanode.max.locked.memory=536870912
174
Add  Save
Restart HDFS using Ambari

You need to re-log on the terminal and verify the configuration.
175
Let us verify the cache configured using the dfsadmin report ; use hdfs user id for
running these commands.
# hdfs dfsadmin -report
Here Used is 0B and configured is 512MB.

176
You can add the directive to determine the used Size.

#hdfs cacheadmin -addDirective -path /user/hdfs/weblog/access_log -pool cPTos
#hdfs cacheadmin -addDirective -path /user/hdfs/weblog/excite-small.log -pool cPTos
Run the report again to determine the cache usage size.

# hdfs dfsadmin -report
177
Now, We are able to store files in the cache pool.

You can verify the pool details:
#hdfs cacheadmin -listPools -stats cPTos
178
This way you can determine the total capacity of cache configured and used.
---------------------- Labs End Here ---------------------------------------------------
179
13. Archival Disk Storage
Using the COLD storage policy

Consider a 1-node cluster in which one disk is configured as archive storage. This node has
two disks and limited computing resources.
1. Attach the archival storage. To set up two data volumes, mount the disks on
separate directories, as shown in the following example:
/hadoop/hdfs/data. (Already exist)

/hadoop/hdfs/archive. (create this using mkdir with root on slave node )
2. Enable a storage policy. The dfs.storage.policy.enabled property is not shown in

the Ambari interface, and you must add it as a custom property to the hdfs-site.xml
file by drilling down from the Dashboard into Configs > Advanced > Custom hdfs-site
> Add Property. The Add Property window is shown in the following figure:
180
Save it. Accept all the default setting.
3. Configure an archive data node..
Navigate to HDFS > Settings > DataNode and add the archival storage volumes to the
data node directories, as shown in the following figure:
/hadoop/hdfs/data
[ARCHIVE]/hadoop/hdfs/archive
181
Save.
After the changes are saved, restart the HDFS.
Determine an existing file using file viewer or hdfs command.
hdfs dfs -ls -R /user/hdfs
182
In our case we will archive the access_log file. If its not there complete the HDFS lab
to store the mention file in the HDFS.
Apply the COLD storage policy to a directory. As the hdfs user, assign the COLD
storage policy to an HDFS directory by running the following commands from the
command line:
su -hdfs
hdfs dfs -mkdir /archives
hdfs dfs -chmod 777 /archives
hdfs storagepolicies -setStoragePolicy -path /archives -policy COLD
After setup is complete, verify the policies by running the following command:
hdfs storagepolicies -getStoragePolicy -path /archives
Output from this command will be similar to the following example:
183
Verify the setup. When you first check the blocks on this archive data node, it
returns 0 because there is no data, as shown in the following figure:
Hdfs  Summary  Quick Links  namenode UI  Data Nodes  Click on the
node else
http://tos.hp.com:50075/datanode.html
And scroll to the volume Information.
Copy a file to the /archives directory, as shown in the following example:

184
hdfs dfs -cp /user/hdfs/weblog/access_log /archives/
hdfs dfs -setrep 1 /archives/access_log
Note: Because this example shows only one archive data node, the replication factor
for the file that was uploaded to archive storage must be set to 1; otherwise, there
will not be enough data nodes for other replicas.
If you check the blocks on this data node again, it now returns a nonzero value that
depends on the size of the uploaded file, as shown in the following figure:
Refresh the datanode page.
This number does not change if you upload files to directories other than /archives.
Let us copy the same file in a different directory.
185
hdfs dfs -cp /user/hdfs/weblog/access_log /tmp
You will observe that there is a increased in no of blocks in the data folder volume
where as archive remain the same i.e if the cold policy is not applied then the data
are stored in the the data folder.
There is an increased in 4 blocks i.e 188 which was 184.
---------------------------- Lab Ends Here --------------------------------------------------
186
14. Erasure Coding
You need to complete the Storage Policy Lab before going ahead with this lab.
As you can see that archive storage is occupying 485.34 mb of storage.
Since this is a cold storage let us apply EC policy so that we can optimize space.
Before configuring EC policies, you can view the list of available erasure coding
policies:
hdfs ec –listPolicies
187
In the previous example, the list of erasure coding policies indicates that RS-6-3-
1024k is already enabled.
Check the block status of the directory before applying erasure-coded on it.
#hdfs fsck /archives
188
Setting an erasure coding policy
The following example shows how you can set the erasure coding policy RS-6-3-1024k
on a particular directory:
hdfs ec -setPolicy -path /archives -policy RS-6-3-1024k
To confirm whether the specified directory has the erasure coding policy applied, run the
hdfs ec -getPolicy command:
189
hdfs ec -getPolicy -path /archives
Take a note of the warning. Setting a policy only affects newly created files, and does
not affect existing files.
Checking the block status on an erasure-coded directory
After enabling erasure coding on a directory, you can check the block status by running
the hdfs fsck command. The following example output shows the status of the erasure-
coded blocks: Which is still not reflected.
hdfs fsck /archives
190
Move the file.

hadoop distcp hdfs://tos.hp.com:8020/archives/access_log
hdfs://tos.hp.com:8020/archives/access_log1
hdfs dfs -cp /user/hdfs/weblog/access_log /archives/access_log_new
191
hdfs ec -enablePolicy -policy XOR-2-1-1024k

hdfs ec -setPolicy -path /archives -policy XOR-2-1-1024k.
Add one more node. – Optional (You need as much nodes as the block and parity group)
hdfs dfs -cp /user/hdfs/weblog/access_log /archives/access_log_new
(When you have three nodes cluster.)
As shown below, replication factor is shown as 1 for the erasure coding folder.
http://tos.hp.com:50070/explorer.html#/archives
192
--------------------------- Lab ends here -------------------------------------------
193
15. Capacity scheduler
In this tutorial we will explore how we can configure YARN Capacity Scheduler using
Ambari.
YARN’s Capacity Scheduler is designed to run Hadoop applications in a shared, multi-

tenant cluster while maximizing the throughput and the utilization of the cluster.
Login to Ambari.
After you Login, you will see the Dashboard.
194
You can drill into a specific service dashboard and configuration.
Let’s dive into YARN dashboard by selecting Yarn from the left-side bar or the drop down
menu.
195
We will start updating the configuration for Yarn Capacity Scheduling policies. Click
on Configs tab and click on Advanced.
196
Next, scroll down to the Scheduler section of the page. The default capacity scheduling
policy just has one queue which is default.
197
Let’s check out the scheduling policy visually. Scroll up to the top of the page click
on SUMMARY and then select ResourceManager UI from the Quick Links section.
198
Once in ResourceManager UI select Queues, you will see a visual representation of the
Scheduler Queue and resources allocated to it.
199
ADJUST THE SCHEDULING POLICY FOR DIFFERENT DEPARTMENTS
Let’s change the capacity scheduling policy to where we have separate queues and policies
for Engineering, Marketing and Support.
Return to YARN Advanced Configs to adjust the Scheduler by navigating to Ambari

Dashboard>YARN>Configs>Advanced>Scheduler
Replace the content for the configurations shown below in the Capacity
Scheduler textbox:
200
yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.root.Engineering.Development.acl_administer_jobs=*
yarn.scheduler.capacity.root.Engineering.Development.acl_administer_queue=*
yarn.scheduler.capacity.root.Engineering.Development.acl_submit_applications=*
yarn.scheduler.capacity.root.Engineering.Development.capacity=20
yarn.scheduler.capacity.root.Engineering.Development.minimumaximum-capacity=100
yarn.scheduler.capacity.root.Engineering.Development.state=RUNNING
yarn.scheduler.capacity.root.Engineering.Development.user-limit-factor=1
yarn.scheduler.capacity.root.Engineering.QE.acl_administer_jobs=*
yarn.scheduler.capacity.root.Engineering.QE.acl_administer_queue=*
yarn.scheduler.capacity.root.Engineering.QE.acl_submit_applications=*
yarn.scheduler.capacity.root.Engineering.QE.capacity=80
yarn.scheduler.capacity.root.Engineering.QE.maximum-capacity=90
yarn.scheduler.capacity.root.Engineering.QE.state=RUNNING
yarn.scheduler.capacity.root.Engineering.QE.user-limit-factor=1
yarn.scheduler.capacity.root.Engineering.acl_administer_jobs=*
yarn.scheduler.capacity.root.Engineering.acl_administer_queue=*
yarn.scheduler.capacity.root.Engineering.acl_submit_applications=*
yarn.scheduler.capacity.root.Engineering.capacity=60
yarn.scheduler.capacity.root.Engineering.maximum-capacity=100
yarn.scheduler.capacity.root.Engineering.queues=Development,QE
yarn.scheduler.capacity.root.Engineering.state=RUNNING
yarn.scheduler.capacity.root.Engineering.user-limit-factor=1
201
yarn.scheduler.capacity.root.Marketing.Advertising.acl_administer_jobs=*
yarn.scheduler.capacity.root.Marketing.Advertising.acl_administer_queue=*
yarn.scheduler.capacity.root.Marketing.Advertising.acl_submit_applications=*
yarn.scheduler.capacity.root.Marketing.Advertising.capacity=30
yarn.scheduler.capacity.root.Marketing.Advertising.maximum-capacity=40
yarn.scheduler.capacity.root.Marketing.Advertising.state=STOPPED
yarn.scheduler.capacity.root.Marketing.Advertising.user-limit-factor=1
yarn.scheduler.capacity.root.Marketing.Sales.acl_administer_jobs=*
yarn.scheduler.capacity.root.Marketing.Sales.acl_administer_queue=*
yarn.scheduler.capacity.root.Marketing.Sales.acl_submit_applications=*
yarn.scheduler.capacity.root.Marketing.Sales.capacity=70
yarn.scheduler.capacity.root.Marketing.Sales.maximum-capacity=80
yarn.scheduler.capacity.root.Marketing.Sales.minimum-user-limit-percent=20
yarn.scheduler.capacity.root.Marketing.Sales.state=RUNNING
yarn.scheduler.capacity.root.Marketing.Sales.user-limit-factor=1
yarn.scheduler.capacity.root.Marketing.acl_administer_jobs=*
yarn.scheduler.capacity.root.Marketing.acl_submit_applications=*
yarn.scheduler.capacity.root.Marketing.capacity=10
yarn.scheduler.capacity.root.Marketing.maximum-capacity=40
yarn.scheduler.capacity.root.Marketing.queues=Sales,Advertising
yarn.scheduler.capacity.root.Marketing.state=RUNNING
yarn.scheduler.capacity.root.Marketing.user-limit-factor=1
yarn.scheduler.capacity.root.Support.Services.acl_administer_jobs=*
yarn.scheduler.capacity.root.Support.Services.acl_administer_queue=*
yarn.scheduler.capacity.root.Support.Services.acl_submit_applications=*
202
yarn.scheduler.capacity.root.Support.Services.capacity=80
yarn.scheduler.capacity.root.Support.Services.maximum-capacity=100
yarn.scheduler.capacity.root.Support.Services.minimum-user-limit-percent=20
yarn.scheduler.capacity.root.Support.Services.state=RUNNING
yarn.scheduler.capacity.root.Support.Services.user-limit-factor=1
yarn.scheduler.capacity.root.Support.Training.acl_administer_jobs=*
yarn.scheduler.capacity.root.Support.Training.acl_administer_queue=*
yarn.scheduler.capacity.root.Support.Training.acl_submit_applications=*
yarn.scheduler.capacity.root.Support.Training.capacity=20
yarn.scheduler.capacity.root.Support.Training.maximum-capacity=60
yarn.scheduler.capacity.root.Support.Training.state=RUNNING
yarn.scheduler.capacity.root.Support.Training.user-limit-factor=1
yarn.scheduler.capacity.root.Support.acl_administer_jobs=*
yarn.scheduler.capacity.root.Support.acl_administer_queue=*
yarn.scheduler.capacity.root.Support.acl_submit_applications=*
yarn.scheduler.capacity.root.Support.capacity=30
yarn.scheduler.capacity.root.Support.maximum-capacity=100
yarn.scheduler.capacity.root.Support.queues=Training,Services
yarn.scheduler.capacity.root.Support.state=RUNNING
yarn.scheduler.capacity.root.Support.user-limit-factor=1
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.queues=Support,Marketing,Engineering
yarn.scheduler.capacity.root.unfunded.capacity=50
203
Click Save and confirm on the dialog box:
204
Changed Capacity Scheduler Config.

In the next two windows that appear click Ok and Proceed Anyway, next you will find a
confirmation window select REFRESH YARN QUEUES
At this point, the configuration is saved but we still need to restart the affected
components by the configuration change as indicated in the orange band below:
205
Also note that there is now a new version of the configuration as indicated by the
green Current label.
Let’s restart the daemons by clicking on the three dots ... next to Services under the
Ambari Stack. Select Restart All Required and wait for the restart to complete.
206
And then go back to the Resource Manager UI and refresh the page.Yup! There’s our new
policy:
207
208
Step 7: Submit a Job to a Specific Queue

Let’s submit a MapReduce job to each new queue
Open Two Terminal by connecting to any of the node.
In the first Terminal – master node
# su - hdfs
# yarn jar $YARN_EXAMPLES/hadoop-mapreduce-examples-3.1.1.3.1.0.0-
78.jar pi -Dmapred.job.queue.name=Training 16 1000
In the Second Terminal – slave node
# su - hdfs
# yarn jar $YARN_EXAMPLES/hadoop-mapreduce-examples-3.1.1.3.1.0.0-
78.jar pi -Dmapred.job.queue.name=QE 16 1000
209
210
7.1. While both jobs are running, refresh the Queue

status page. (It may take
a minute for both jobs to run long enough to show up in the queues, so
refresh the page often until they do):
Click on QE you can see that one job is running in this Queue.
211
Keep on refershing the above page to determine the % of resources use during the execution
of the jobs by respective Queue.
7.2. You should see resources being used in both the Support and Engineering queues.
212
RESULT: You just defined two queues for the Capacity Scheduler, configured
specific capacities for each queue, and submitted a job to each queue.
-------------------------------------------- LAB ENDS HERE ----------------------------------------
213
16. ROLLING BACK CONFIGURATION SETTING (Optional )

You can also rollback to the previous set of configurations.
Return to Ambari Dashboard then select Config History and select YARN:
Finally, select Make Current and re-start services one last time.
214
215
17. NameNode HA
Objective: To configure and verify NameNode High Availability using Ambari.
Successful Outcome: Your cluster will have a Standby NameNode along with Active
NameNode.
Before You Begin: Open the Ambari UI.
Prerequisite:
You have added two nodes in the hadoop cluster and the services are up as shown below:
216
You require three nodes for configuring Namenode HA. Hence let us add another node. i.e
slavea
Server Service
Master Ambari server , Ambari agent
Slave Ambari agent
Slavea Ambari agent New Addition
217
Add Nodea i.e Slavea.
Refer the earlier Lab - Adding Node to the cluster. Summary is given as shown below:
For clarity let us rename the machine name as follows:
IP Name Remarks Services

10.10.20.20 master First Machine Master and slave services
10.10.20.21 slave 2 Machine NodeManager Data Node
10.10.20.25 slavea New Machine Node Manager and Data

Node
Rename the host as.tos.slavea.com using nmtui.
You can verify after typing : bash and then hostname
218
Ensure to install java on the host. You can refer the first Lab for this.
Install the java mysql connector

yum install mysql-connector-java*
cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-agent/tmp/
Update /etc/hosts on all three machines with the following lines: Ensure that you update with
your IPs appropriately. Append it don’t delete other entries.
# vi /etc/hosts (for master AND slave)

10.10.20.20 tos.master.com
219
10.10.20.24 tos.slave.com
10.10.20.25 tos.slavea.com
Enable SSH access between master and slavea

Execute from the master node: Stop the forewalld on the slave node a (systemctl stop
firewalld)
$ ssh-copy-id -i $HOME/.ssh/id_rsa.pub [email protected]
Log in to Ambari (tos.master.com), click on Hosts and choose Add New Hosts from
the Actions menu.
220
Select the following options and ensure that other options are leave as default.
221
222
In step Assign Slaves and Clients I define my node to be a DataNode and has a
NodeManager installed as well
You should have 3 nodes in the dashboard as shown below.
Log on Ambari server  Dashboard  Hosts
223
As shown above you need to have three node clusters before proceeding ahead for this lab.
Install zookeeper on two slave nodes: slave and slavea node.
Click on Host  Hostname  Add  Zookeper
224
225
Accept the default value and let us proceed.

At the end of the above steps you should have three nodes zookeeper.
Step 2: Enable NameNode HA

1. In Ambari Web, browse to Services > HDFS > Summary.
2. Click Actions, then click Enable NameNode HA.
226
The Enable NameNode HA wizard launches.
Enter HACluster as the Nameservice ID for your NameNode HA cluster. (The name must
consist of one word and not have any special characters.) Click the Next button:
2.1. Choose slavea as the Additional NameNode:
227
228
2.2. Click the Next button.
2.3. Notice that a Secondary NameNode is not allowed if you have NameNode
HA configured. Ambari takes care of this for you, as you can see on the
Review step of the wizard:
2.4. Click the Next button to continue the wizard.
Step 3: Perform the Manual Steps

3.1. Notice the Enable NameNode HA Wizard is requiring you to perform
some manual steps before you are able to continue. Start by SSH--‐ing into
229
node1.
3.2. Put the NameNode in Safe Mode:
# sudo su -l hdfs -c 'hdfs dfsadmin -safemode enter'
3.3. Create a Checkpoint:
# sudo su -l hdfs -c 'hdfs dfsadmin -saveNamespace'
3.4. Once Ambari recognizes that your cluster is in Safe Mode and a
Checkpoint has been made, you will be able to click the Next button.
230
Step 4: Wait for the Configuration

4.1. At this point, Ambari will stop all services, install the necessary
components, and restart the services. Wait for these tasks to complete:
231
4.2. Once all the tasks are complete, click the Next button.
Step 5: Initialize the JournalNodes

5.1. On node1, enter the command shown in the wizard to initialize
the JournalNodes:
# sudo su -l hdfs -c 'hdfs namenode -initializeSharedEdits'
5.2. Once Ambari determines that the JournalNodes are initialized, you
will be able to click the Next button:
Step 6: Start the Components

6.1. In the next step, Ambari will start ZooKeeper and the NameNode service. Click
Next when its complete:
Step 7: Initialize NameNode HA Metadata

7.1. On master, enter the command shown in the wizard to configure
the metadata for automatic failover:
# sudo su -l hdfs -c 'hdfs zkfc -formatZK'
7.2. On slavea, run the command to initialize the metadata for the
new NameNode:
# sudo su -l hdfs -c 'hdfs namenode -bootstrapStandby'

7.3. Click the Next button to continue.
Step 8: Wait for the Wizard to Finish

8.1. In this final step, the wizard will start all required services and delete
the Secondary NameNode:
8.2. Click the Done button when all the tasks are complete.
Step 9: Verify the Standby NameNode

9.1. Go to the HDFS service page. You should see an Active NameNode and a
Standby NameNode:
Step 10: Test NameNode HA.

10.1. Go the Host page of slavea in Ambari. (Here in my case slavea is the master)
10.2. Next to the Active NameNode component, select Stop from the Action
menu:
10.3. Go back the HDFS page in Ambari. Notice the Standby

NameNode has become the Active NameNode:
10.4. Now start the stopped NameNode again, and you will notice that it
becomes a Standby NameNode:
RESULT: You now have NameNode HA configured on your cluster, and you
have also verified that the HA works when one of the NameNodes stops.
-------------------------------------------- Lab ends here ----------------------------------------------

18. ResourceManager high availability
To access the wizard and enable ResourceManager high availability:

You can configure high availability for ResourceManager by using the Enable
ResourceManager HA wizard.
You must have at least three nodes in the cluster:

 hosts in your cluster
Start Master, slave and slavea nodes.
 Apache ZooKeeper servers running on all nodes.
Start the following in the master node:

Namenode, DataNode, Resource Manager and Node Manager Services.
In all nodes,
Datanode, Zookeeper, Zkfc & Journal Node wherever it’s installed.
In slavea
Namenode
At this point we have only one Resource Manager configured in the cluster.
Tos.Tech | http://thinkopensource.in
HDFS Services should be up as shown below:
YARN Services should be up too.
1. In Ambari Web, browse to Services > YARN > Summary.

2. Select Service Actions and choose Enable ResourceManager HA.
The Enable ResourceManager HA wizard launches, describing a set of automated
and manual steps that you must take to set up ResourceManager high availability.
3. On Get Started, read the overview of enabling ResourceManager HA.
Click Next to proceed.
4. On Select Host (Slavea), accept the default selection or choose an available host.
Click Next to proceed.

5. On Review Selections, expand YARN if necessary, to review all the configuration
changes proposed for YARN.
Click Next to approve the changes and start automatically configuring ResourceManager
HA.
6. On Configure Components, click Complete when all the progress bars finish
tracking.
At the end you should have the following two nodes of RM
As you can see one will be in active and other will be in stand by.
Test the Fail Over.

Submit a job.
Let’s submit a MapReduce job to the cluster
In the Terminal – master node
# su - hdfs
# yarn jar $YARN_EXAMPLES/hadoop-mapreduce-examples-3.1.1.3.1.0.0-78.jar pi -
Dmapred.job.queue.name=Training 6 10
Stop the Primary RM after the job execution started.

Determine the RM node which is primary, execute the following command and the one with
the state active is the primary node.
#su - yarn
yarn rmadmin -getServiceState rm1
In my case the rm1 is the primary RM. (Yarn  Config  Custom yarn-site.xml)
#rm1 is the primary resource manager.
Stop the RM service of master node. When the terminal display the below text
Using the dashboard  Yarn  Active  STop
Failing Over RM node message will be displayed in the console as shown below:
Verify that Job proceed with the secondary RM

After sometimes it should be failed over to rm2
You can verify from the dashboard also that rm2 is the primary resource manager now.
Now the job will be orchestrated by the new primary RM i.e rm2 and we don’t need to
resubmit the job to the cluster.
You can also determine the status of the rm2 i.e slavea using yarn command.
Job will be completed after a few minutes depending on the resources.
Let us start the Failed RM i.e rm1 ; master node.
Let us check which is primary now.

This mean that the current active node will be the primary Resource Manager till it get
failed over although the earlier primary node comes up.
-------------------------------------------- Lab ends here ----------------------------------------------
19. Tuning and Debugging HDP
Case Study to debug and resolve job related issues running in Hadoop cluster. You can’t
execute any job that exceeds the Virtual memory demand then the configure in the
configuration file.
If the virtual memory usage exceeds more than the allowed configured memory then the
container will be killed and job will failed.
Let us enable the flag in custom yarn-site.xml file so that Node Manager can monitor the
virtual memory usage of the cluster. i.e yarn.nodemanager.vmem-check-enabled = true
Dashboard  Yarn  Config  Advance
Accept all warning and default setting to complete the configuration.

You need to restart the following services to get the setting affected.
Resource manager, Node manager services in all the applicable nodes.
Submit the following job to the cluster.
# su - hdfs
#yarn jar $YARN_EXAMPLES/hadoop-mapreduce-examples-3.1.1.3.1.0.0-78.jar pi 16
1000
After some time the Job will failed with the following errors.
Container Physical Memory consumption at this juncture: Virtual memory usage is beyond
the permissible limit.
Current usage: 107.5 MB of 206 MB physical memory used …
Physical Memory allocated is 2536 MB i.e 2.4 GB
1.9 GB of 824 MB virtual memory used ( Virtual memory usage exceeds that of the limit).
Killing container.
Observation:
Open the file and observe the virtual to physical memory allowed ratio. It’s 4 times here for
each map container.
#vi /etc/hadoop/3.1.0.0-78/0/yarn-site.xml
yarn.nodemanager.vmem-pmem-ratio = 4 times
the ("mapreduce.map.memory.mb") is set to 206MB then the total allowed virtual memory
is 4 * 206 =824MB.
#vi /etc/hadoop/3.1.0.0-78/0/mapred-site.xml
However as shown in the below log 1.9 GB virtual memory is demanded then the allowed
824 Gb configured. Hence the job failed.
You can verify from the log. This error is due to the overall consumption of virtual memory
which is more than the allocated allowed virtual memory. How do we resolve this? One way
is to increase the Physical memory and raised the allowed virtual memory ratio. Another
way is to disable the validation of the virtual memory. Which we will disable it!
Concepts:
NodeManager can monitor the memory usage (virtual and physical) of the container. If its
virtual memory exceeds “yarn.nodemanager.vmem-pmem-ratio” times the
"mapreduce.reduce.memory.mb" or "mapreduce.map.memory.mb", then the container will
be killed if “yarn.nodemanager.vmem-check-enabled” is true.
Solution:
yarn.nodemanager.vmem-check-enabled should be false and restart the cluster services i.e
Nodemanager and Resource Manager
Then resubmit the job all over again. Do not update the xml files directly. All changes have
to be done through the Ambari UI only.
-------------------------------------------- Lab ends here ------------------------------------------
20. Managing YARN Config
Set up Queues
Capacity Scheduler queues can be set up in a hierarchy that reflects the database structure,
resource requirements, and access restrictions required by the various organizations,
groups, and users that utilize cluster resources.
The fundamental unit of scheduling in YARN is a queue. The capacity of each queue
specifies the percentage of cluster resources that are available for applications submitted to
the queue.
For example, suppose that a company has three organizations: Engineering, Support, and
Marketing. The Engineering organization has two sub-teams: Development and QA. The
Support organization has two sub-teams: Training and Services. And finally, the Marketing
organization is divided into Sales and Advertising. The following image shows the queue
hierarchy for this example:
Each child queue is tied to its parent queue with the yarn.scheduler.capacity.<queue-
path>.queues configuration property in the capacity-scheduler.xml file. The top-level
"support", "engineering", and "marketing" queues would be tied to the "root" queue as
follows:
Property: yarn.scheduler.capacity.root.queues
Value: support,engineering,marketing
Example:
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>support,engineering,marketing</value>
<description>The top-level queues below root.</description>
</property>
Similarly, the children of the "support" queue would be defined as follows:
Property: yarn.scheduler.capacity.support.queues
Value: training,services
Example:
<property>
<name>yarn.scheduler.capacity.support.queues</name>
<value>training,services</value>
<description>child queues under support</description>
</property>
The children of the "engineering" queue would be defined as follows:
Property: yarn.scheduler.capacity.engineering.queues
Value: development,qa
Example:
<property>
<name>yarn.scheduler.capacity.engineering.queues</name>
<value>development,qa</value>
<description>child queues under engineering</description>
</property>
And the children of the "marketing" queue would be defined as follows:
Property: yarn.scheduler.capacity.marketing.queues
Value: sales,advertising
Example:
<property>
<name>yarn.scheduler.capacity.marketing.queues</name>
<value>sales,advertising</value>
<description>child queues under marketing</description>
</property>
Set User Limits

Set a minimum percentage of resources allocated to each leaf queue user.
 The minimum-user-limit-percent property can be used to set the minimum percentage of
resources allocated to each leaf queue user. For example, to enable equal sharing of the
"services" leaf queue capacity among five users, you would set the minimum-user-limit-
percent property to 20%:
Property: yarn.scheduler.capacity.root.support.services.minimum-user-limit-percent
Value: 20
This setting determines the minimum limit that any user’s share of the queue capacity
can shrink to. Irrespective of this limit, any user can come into the queue and take more
than his or her allocated share if there are idle resources available.
The following table shows how the queue resources are adjusted as users submit jobs to a
queue with a minimum-user-limit-percent value of 20%:
For example, the following properties would set the root acl_submit_applications value to
"" (space character) to block access to all users and groups, and also restrict access to its
child "support" queue to the users "sherlock" and "pacioli" and the members of the "cfo-
group" group:
Each child queue is tied to its parent queue with the yarn.scheduler.capacity.<queue-
path>.queues configuration property in the capacity-scheduler.xml file. The top-level
"support", "engineering", and "marketing" queues would be tied to the "root" queue as
follows:
<property>
<name>yarn.scheduler.capacity.root.acl_submit_applications</name>
<value> </value>
</property>
<property>
<name>yarn.scheduler.capacity.root.support.acl_submit_applications</name>
<value>sherlock,pacioli cfo-group</value>
</property>
Types of node labels:

Exclusive – In this type of node labels, only associated/mapped queues can access the
resources of node label.
Non Exclusive(sharable) – If resources are not in use for this node label then it can be
shared with other running applications in a cluster.
.
Configuring node labels:
.
Step 1: Create required directory structure on HDFS
Note – You can run below commands from any of the hdfs client.
1. sudo su hdfs
2. hadoop fs -mkdir -p /yarn/node-labels
3. hadoop fs -chown -R yarn:yarn /yarn
4. hadoop fs -chmod -R 700 /yarn
.
Step 2: Make sure that you have user directory for ‘yarn’ user on HDFS, if not then please
create it using below commands
Note – You can run below commands from any of the hdfs client.
1. sudo su hdfs
2. hadoop fs -mkdir -p /user/yarn
3. hadoop fs -chown -R yarn:yarn /user/yarn
4. hadoop fs -chmod -R 700 /user/yarn
.
Step 3: Configure below properties in yarn-site.xml via Ambari UI. If you don’t have Ambari
UI, please add it manually to /etc/hadoop/conf/yarn-site.xml and restart required services.
1. yarn.node-labels.enabled=true
2. yarn.node-labels.fs-store.root-dir=hdfs://<namenode-host>:<namenode-rpc-port>/<co
mplete-path_to_node_label_directory>
Note – Please restart required services after above configuration changes!

.
Step 4: Create node labels using below commands
1. sudo -u yarn yarn rmadmin -addToClusterNodeLabels "<node-label1>(exclusive=<true|f

alse>),<node-label2>(exclusive=<true|false>)"
For example, to add 2 node labels x and y:
1. sudo -u yarn yarn rmadmin -addToClusterNodeLabels "x(exclusive=true),y(exclusive=fal

se)"
You can verify if node labels have been created by looking at Resource manager UI under
‘Node Lables’ option in the left pane or you can also run below command on any of the Yarn
client
1. yarn cluster --list-node-labels
Sample output:
1. [yarn@prodnode1 ~]$ yarn cluster --list-node-labels

2. 16/12/14 15:45:56 INFO impl.TimelineClientImpl: Timeline service address: http://prod
node3.openstacklocal:8188/ws/v1/timeline/
3. 16/12/14 15:45:56 INFO client.RMProxy: Connecting to ResourceManager at prodnode3.
openstacklocal/172.26.74.211:8050
4. Node Labels: <x:exclusivity=true>,<y:exclusivity=false>
.
Step 5: Allocate node labels to the node managers using below command:
1. sudo -u yarn yarn rmadmin -replaceLabelsOnNode "<node-manager1>:<port>=<node-l

abel1> <node-manager2>:<port>=<node-label2>"
Example:
1. sudo -u yarn yarn rmadmin -replaceLabelsOnNode "prodnode1.openstacklocal=x prodno

de2.openstacklocal=y"
Note – Don’t worry about port if you have only one node manager running per host.
.
Step 6: Map node labels to the queues:
I have created 2 queues ‘a’ and ‘b’ in such a way that, queue ‘a’ can access nodes with label
‘x’ and ‘y’ where queue ‘b’ can only access the nodes with label ‘y’. By default, all the queues
can access nodes with ‘default’ label.
Below is my capacity scheduler configuration:
1. yarn.scheduler.capacity.maximum-am-resource-percent=0.2
2. yarn.scheduler.capacity.maximum-applications=10000
3. yarn.scheduler.capacity.node-locality-delay=40
4. yarn.scheduler.capacity.queue-mappings-override.enable=false
5. yarn.scheduler.capacity.root.a.a1.accessible-node-labels=x,y
6. yarn.scheduler.capacity.root.a.a1.accessible-node-labels.x.capacity=30
7. yarn.scheduler.capacity.root.a.a1.accessible-node-labels.x.maximum-capacity=100
8. yarn.scheduler.capacity.root.a.a1.accessible-node-labels.y.capacity=50
9. yarn.scheduler.capacity.root.a.a1.accessible-node-labels.y.maximum-capacity=100
10. yarn.scheduler.capacity.root.a.a1.acl_administer_queue=*
11. yarn.scheduler.capacity.root.a.a1.acl_submit_applications=*
12. yarn.scheduler.capacity.root.a.a1.capacity=40
13. yarn.scheduler.capacity.root.a.a1.maximum-capacity=100
14. yarn.scheduler.capacity.root.a.a1.minimum-user-limit-percent=100
15. yarn.scheduler.capacity.root.a.a1.ordering-policy=fifo
16. yarn.scheduler.capacity.root.a.a1.state=RUNNING
17. yarn.scheduler.capacity.root.a.a1.user-limit-factor=1
18. yarn.scheduler.capacity.root.a.a2.accessible-node-labels=x,y
19. yarn.scheduler.capacity.root.a.a2.accessible-node-labels.x.capacity=70
20. yarn.scheduler.capacity.root.a.a2.accessible-node-labels.x.maximum-capacity=100
21. yarn.scheduler.capacity.root.a.a2.accessible-node-labels.y.capacity=50
22. yarn.scheduler.capacity.root.a.a2.accessible-node-labels.y.maximum-capacity=100
23. yarn.scheduler.capacity.root.a.a2.acl_administer_queue=*
24. yarn.scheduler.capacity.root.a.a2.acl_submit_applications=*
25. yarn.scheduler.capacity.root.a.a2.capacity=60
26. yarn.scheduler.capacity.root.a.a2.maximum-capacity=60
27. yarn.scheduler.capacity.root.a.a2.minimum-user-limit-percent=100
28. yarn.scheduler.capacity.root.a.a2.ordering-policy=fifo
29. yarn.scheduler.capacity.root.a.a2.state=RUNNING
30. yarn.scheduler.capacity.root.a.a2.user-limit-factor=1
31. yarn.scheduler.capacity.root.a.accessible-node-labels=x,y
32. yarn.scheduler.capacity.root.a.accessible-node-labels.x.capacity=100
33. yarn.scheduler.capacity.root.a.accessible-node-labels.x.maximum-capacity=100
34. yarn.scheduler.capacity.root.a.accessible-node-labels.y.capacity=50
35. yarn.scheduler.capacity.root.a.accessible-node-labels.y.maximum-capacity=100
36. yarn.scheduler.capacity.root.a.acl_administer_queue=*
37. yarn.scheduler.capacity.root.a.acl_submit_applications=*
38. yarn.scheduler.capacity.root.a.capacity=40
39. yarn.scheduler.capacity.root.a.maximum-capacity=40
40. yarn.scheduler.capacity.root.a.minimum-user-limit-percent=100
41. yarn.scheduler.capacity.root.a.ordering-policy=fifo
42. yarn.scheduler.capacity.root.a.queues=a1,a2
43. yarn.scheduler.capacity.root.a.state=RUNNING
44. yarn.scheduler.capacity.root.a.user-limit-factor=1
45. yarn.scheduler.capacity.root.accessible-node-labels=x,y
46. yarn.scheduler.capacity.root.accessible-node-labels.x.capacity=100
47. yarn.scheduler.capacity.root.accessible-node-labels.x.maximum-capacity=100
48. yarn.scheduler.capacity.root.accessible-node-labels.y.capacity=100
49. yarn.scheduler.capacity.root.accessible-node-labels.y.maximum-capacity=100
50. yarn.scheduler.capacity.root.acl_administer_queue=*
51. yarn.scheduler.capacity.root.b.accessible-node-labels=y
52. yarn.scheduler.capacity.root.b.accessible-node-labels.y.capacity=50
53. yarn.scheduler.capacity.root.b.accessible-node-labels.y.maximum-capacity=100
54. yarn.scheduler.capacity.root.b.acl_administer_queue=*
55. yarn.scheduler.capacity.root.b.acl_submit_applications=*
56. yarn.scheduler.capacity.root.b.b1.accessible-node-labels=y
57. yarn.scheduler.capacity.root.b.b1.accessible-node-labels.y.capacity=100
58. yarn.scheduler.capacity.root.b.b1.accessible-node-labels.y.maximum-capacity=100
59. yarn.scheduler.capacity.root.b.b1.acl_administer_queue=*
60. yarn.scheduler.capacity.root.b.b1.acl_submit_applications=*
61. yarn.scheduler.capacity.root.b.b1.capacity=100
62. yarn.scheduler.capacity.root.b.b1.maximum-capacity=100
63. yarn.scheduler.capacity.root.b.b1.minimum-user-limit-percent=100
64. yarn.scheduler.capacity.root.b.b1.ordering-policy=fifo
65. yarn.scheduler.capacity.root.b.b1.state=RUNNING
66. yarn.scheduler.capacity.root.b.b1.user-limit-factor=1
67. yarn.scheduler.capacity.root.b.capacity=60
68. yarn.scheduler.capacity.root.b.maximum-capacity=100
69. yarn.scheduler.capacity.root.b.minimum-user-limit-percent=100
70. yarn.scheduler.capacity.root.b.ordering-policy=fifo
71. yarn.scheduler.capacity.root.b.queues=b1
72. yarn.scheduler.capacity.root.b.state=RUNNING
73. yarn.scheduler.capacity.root.b.user-limit-factor=1
74. yarn.scheduler.capacity.root.capacity=100
75. yarn.scheduler.capacity.root.queues=a,b
-------------------------------------------- Lab ends here -------------------------------------------
21. Miscelleanous
Enable User Home Directory Creation.
1. Edit the ambari-properties file using a command line editor (vi, in this example).
vi /etc/ambari-server/conf/ambari.properties
2. Add the following property:
ambari.post.user.creation.hook.enabled=true
3. Add the script path to the ambari properties file:
ambari.post.user.creation.hook=/var/lib/ambari-server/resources/scripts/post-user-creation-hook.sh
4. Restart Ambari server.
ambari-server restart
Default scheduler:
yarn.scheduler.capacity.default.minimum-user-limit-percent=100
yarn.scheduler.capacity.maximum-am-resource-percent=0.2
yarn.scheduler.capacity.maximum-applications=10000
yarn.scheduler.capacity.node-locality-delay=40
yarn.scheduler.capacity.resource-
calculator=org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator
yarn.scheduler.capacity.root.accessible-node-labels=*
yarn.scheduler.capacity.root.acl_administer_queue=*
yarn.scheduler.capacity.root.acl_submit_applications=*
yarn.scheduler.capacity.root.capacity=100
yarn.scheduler.capacity.root.default.acl_administer_jobs=*
yarn.scheduler.capacity.root.default.acl_submit_applications=*
yarn.scheduler.capacity.root.default.capacity=100
yarn.scheduler.capacity.root.default.maximum-capacity=100
yarn.scheduler.capacity.root.default.state=RUNNING
yarn.scheduler.capacity.root.default.user-limit-factor=1
yarn.scheduler.capacity.root.queues=default
yarn.scheduler.capacity.schedule-asynchronously.enable=true
yarn.scheduler.capacity.schedule-asynchronously.maximum-threads=1
yarn.scheduler.capacity.schedule-asynchronously.scheduling-interval-ms=10

Tutorial-HDP-Administration V III

Uploaded by

Copyright:

Available Formats

Tutorial-HDP-Administration V III

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Tutorial-HDP-Administration V III

Uploaded by

Copyright:

Available Formats

1 Big Data - Admin Course

Table of Contents ...................................................................................................... 1

17. NameNode HA ................................................................................................... 216

All software will be in D:\\Software folder of your desktop.

Action: Start virtual machine.

Resolution (#vi /etc/fstab)

Reboot the machine to make the changes effect.

Goal: You will install Ambari server on the host.

#echo "export JAVA_HOME=/YARN/jdk/" > /etc/profile.d/java.sh

Include in .bashrc (vi ~/.bashrc)

yum install ambari-server

Set Up the Ambari Server

6. Select n at Enter advanced database configuration to use the default, embedded

Start the Ambari Server

 Run the following command on the Ambari Server host:

 To check the Ambari Server processes:

If any error as shown below occurred during start up:

2019-04-13 20:25:51,247 ERROR - Current database store version is not compatible

Then configure the ambari set up again.

Log on to Apache Ambari

#cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-agent/tmp/

-------------------------------------------- Lab ends here ---------------------------------------------

3. Installing a HDP Cluster

Install MYSQL Server using root by opening a terminal.

Set the mysql root user id password to Life2134.

From the Ambari Welcome page, choose Launch Install Wizard.

Name Your Cluster

Name it: myhdpcluster

Select the HDP version to be installed i.e

Select the remote repository to download the required software:

2. Open a terminal and perform the following:

Enter the hostname of the VM i.e hp.tos.com

It will take some times to highlight the Next button

Accept the default settings; ensure that NFS Gateway is selected.

All passwords should be Life2134 except the following.

Select default setting after this screen and Go to Next steps.

Click on Ranger Tab

Enable the DB setup for ambari.

Set the entire password in this console to Life2134. – Ranger Tab

Verify the test connection and it should be Ok as shown above.

On the RANGER KMS tab,

Supply the db host as localhost and enter password as Life2134.

(Optional-)Reset the password setting as shown below in the customization advance

2019-10-15 03:29:04,579 - File['/usr/lib/ambari-agent/DBConnectionVerification.jar'] {'content':

2019-10-15 03:29:04,606 - Not downloading the file from http://tos.hp.com:8080/resources/DBConnectionVerification.jar,

2019-10-15 03:29:04,638 - File['/var/lib/ambari-agent/tmp/mysql-connector-java.jar'] {'content':

2019-10-15 03:29:04,639 - Downloading the file from http://tos.hp.com:8080/resources/mysql-connector-java.jar

Command failed after 1 tries

2019-06-09 14:04:54,384 - Retrying after 10 seconds. Reason: Execution of

Verify the safemode:

# sudo -u hdfs hdfs dfsadmin -safemode get

#sudo -u hdfs hdfs dfsadmin -safemode leave

Task to stop or start any services.

Hive Issues: (Optional) – unable to start Hive Metastore

Underlying cause: java.sql.SQLException : Access denied for user 'hive'@'tos.hp.com'

4. Understanding YARN Config

Start the Ambari Server

 Run the following command on the Ambari Server host:

 To check the Ambari Server processes: