Professional Documents
Culture Documents
Hadoop Architecture Exercise
Hadoop Architecture Exercise
Hadoop Fundamentals
Unit 2: Hadoop Architecture
IBM Software
Contents
LAB 1
Contents
Page 3
IBM Software
Lab 1
Hadoop Architecture
The overwhelming trend towards digital services, combined with cheap storage, has generated
massive amounts of data that enterprises need to effectively gather, process, and analyze. Data
analysis techniques from the data warehouse and high-performance computing communities
are invaluable for many enterprises, however often times their cost or complexity of scale-up
discourages the accumulation of data without an immediate need. As valuable knowledge may
nevertheless be buried in this data, related scaled-up technologies have been developed.
Examples include Googles MapReduce, and the open-source implementation, Apache Hadoop.
Hadoop is an open-source project administered by the Apache Software Foundation. Hadoops
contributors work for some of the worlds biggest technology companies. That diverse,
motivated community has produced a collaborative platform for consolidating, combining and
understanding data. After completing this hands-on lab, youll be able to:
Page 4
Username
Password
root
password
Linux
biadmin
biadmin
Unit 4
IBM Software
1.1
Getting Started
First start your Quick Start image. You should allocate at least 5 gig of memory to your image.
Hands-on-Lab
Page 5
IBM Software
__4.
Page 6
You need to start the BigInsights components. With this image, you have two options.
One option is to use the icon that was placed on the desktop. But that icon is unique to
the Quick Start image. So start the components in the more traditional way.
Unit 4
IBM Software
__6.
1.2
Hadoop Distributed File System (HDFS) allows user data to be organized in the form of files and
directories. It provides a command line interface called FS shell that lets a user interact with the
data in HDFS accessible to Hadoop MapReduce programs.
There are two ways that you can interact with HDFS:
1. You can use the command-line approach and invoke the File System (fs) shell.
2. You can use the BigInsights Console.
In this exercise you will work with both.
__1.
Start with the command to list files and directories. In your command window,
type the following:
hadoop fs -ls /
__2.
Hands-on-Lab
Now list the files that are in biadmins HDFS home directory .
Page 7
IBM Software
__3.
__5.
Create a file in the Linux home directory for biadmin. From a command line,
execute the following commands. (ctrl-c means to press the Ctrl key and then the c key.)
cd ~
cat > myfile.txt
this is some data
in my file
ctrl-c
Page 8
Unit 4
IBM Software
__6.
Next upload this newly created file to the test directory that you just created.
__8.
__9.
You can pipe (using the | character) any HDFS command to be used with the
Linux shell. For example, you can easily use grep with HDFS by doing the following.
hadoop fs -ls /user/biadmin ! grep test
Hands-on-Lab
Page 9
IBM Software
__10.
To find the size of a particular file, like myfile.txt, execute the following:
__11.
__12.
Page 10
Unit 4
IBM Software
__13.
Remember that you can always use the -help parameter to get more help.
hadoop fs -help
hadoop fs -help du
1.3
BigInsights Console
__1.
On the Quick Start Edition vmware image desktop, you can click an icon to start
the BigInsights Console. Once again this is unique to this image. In real life, you would
start Firefox and type in a URL of http://<host_name>:8080 where <host_name> is the
host where the BigInsights Console is running.
For this exercise, double-click the WebConsole icon on the desktop.
Hands-on-Lab
Page 11
IBM Software
__2.
Log in with a User name of biadmin and the password that you specified for
biadmin. You should be taken to the Welcome tab which provides shortcuts to working
with the console.
__3.
Click the Cluster Status tab. Here you can see the status of each component. On
the left side, click Nodes. Here you can see the status of each node in the cluster, add
and remove nodes and services. Selecting any of the components on the left, shows you
detail information about that component.
Click Map/Reduce. On the right side, you can start and stop the JobTracker and any of
the TaskTrackers. You also can see the host and port number for the JobTracker and
each TaskTracker running on each node. In our case there is only a single node.
Page 12
Unit 4
IBM Software
__4.
Open a new tab in Firefox by clicking on the plus sign tab. Type is a URL of
bivm.ibm.com:50060. This is the web interface that comes with Hadoop.
Hands-on-Lab
Page 13
IBM Software
__5.
tab
In Firefox, close the TaskTracker tab and return to IBM InfoSphere BigInsights
__6.
In the BigInsights Console, click the Files tab. (I have found on the Quick Start
Edition image that there can be a problem having the information on the Files tab
displayed properly. It appears that if I first display a tab other than the Files tab or the
BigSheets tab, my display looks as follows. If you are experiencing the same problem,
my suggestion is to ether click the BigSheets tab and then the Files tab or log out of the
console, log back in and select the Files tab as the first thing. )
Page 14
Unit 4
IBM Software
__7.
Expand the user->biadmin directories. You should be able to see the test
directory that you created earlier.
__8.
Expand the test directory and select myfile.txt. Note that the contents of the file
are displayed on the right side. You can even edit it.
__9.
Become familiar with the functions provided by the icons at the top of the pane.
Simply move your cursor over each icon and view the popup help.
__10.
Select the biadmin directory. Then click the Create Directory icon.
__11.
Hands-on-Lab
Page 15
IBM Software
__12.
Select the newly created ConsoleLab directory and click the Rename icon and
change the directory name to ConsoleLabTest.
Page 16
Unit 4
IBM Software
__13.
__14.
Click the Browse pushbutton. Navigate to the home directory for biadmin. Select
myfile.txt and click Open.
Hands-on-Lab
Page 17
IBM Software
__15.
myfile.txt has been added to the upload list. You could repeat this process and add
additional files to the upload list. Click OK.
Page 18
Unit 4
IBM Software
__16.
Select myfile.txt in the ConsoleLabText directory. Click the Remove icon. Click
Yes on the confirmation dialog.
__17.
Select myfile.txt in the test directory. Then select the Move icon.
__18.
Hands-on-Lab
Page 19
IBM Software
__19.
Select myfile.txt in the ConsoleLabTest directory and click the Set permissions
icon. Here you can make changes to ownership and permissions. For this exercise, no
changes are required.
__20.
__21.
__22.
Page 20
Unit 4
IBM Software
__23.
Close the shell dialog. Then click the Refresh icon. You will probably then get a
message that your file no longer exists. Close that dialog.
1.4
Summary
Congratulations! You're now familiar with the Hadoop Distributed File System. You now know how to
manipulate files within by using the terminal and the BigInsights Console. You may move on to the next
unit.
Hands-on-Lab
Page 21
NOTES
NOTES