Big Data and Business Analytics: Lab Manual
Big Data and Business Analytics: Lab Manual
LAB MANUAL
Regulations : R16
Semester : VII
Branch : IT
Prepared by
1. PROGRAM OUTCOMES:
2
2. PROGRAM SPECIFIC OUTCOMES (PSOs):
Problem-Solving Skills: The ability to apply standard practices and strategies in software
PSO – II project development using open-ended programming environments to deliver a quality product
for business success.
Program
WEEK.N Program Outcomes Specific
Experiment
O Attained Outcomes
Attained
3
HIVE PO1, PO2, PO3, PO4,
11. PSO1, PSO2
PO5, PO12
HIVE OPERATIONS PO1, PO2, PO3, PO4,
12. PSO1, PSO2
PO5, PO12
Program
Specific
Course Program Outcomes (POs)
Outcomes
Objectives (PSOs)
(COs)
PSO1 PSO PSO3
PO1 PO2 PO3 PO4 PO5 PO6 PO7 PO8 PO9 PO10 PO11 PO12
2
√ √ √ √
I
√ √ √ √ √
II
√ √ √ √ √
III
√ √ √ √ √
IV
4
5. SYLLABUS:
Implementing the basic commands of LINUX Operating System – File/Directory creation, deletion, update operations.
5
Implement the following file management tasks in Hadoop:
i. Adding files and directories
ii. Retrieving files
iii. Deleting files
Hint: A typical Hadoop workflow creates data files (such as log files) elsewhere and copies them into HDFS using one of the
above command line utilities.
Run a basic word count Map Reduce program to understand Map Reduce Paradigm.
Week-6 MAPREDUCE PROGRAM 2
Write a Map Reduce program that mines weather data. Hint: Weather sensors collecting data every hour at many locations
across the globe gather a large volume of log data, which is a good candidate for analysis with Map Reduce, since it is semi
structured and record-oriented
Week-7 MAPREDUCE PROGRAM 3
Implement matrix multiplication with Hadoop Map Reduce.
Week-8 PIG LATIN LANGUAGE – PIG
Installation of PIG.
Reference Books:
Jay Liebowitz, ―Big Data And Business Analytics Laboratory, CRC Press.
6. INDEX:
PAGE
S.NO LIST OF EXPERIMENTS
NO
WEEK-1 INSTALL VMWARE
Installation of VMWare to setup the Hadoop environment and its ecosystems. 10
HADOOP MODES
Perform setting up and Installing Hadoop in its three operating modes.
WEEK-2 i. Standalone.
17
a ii. Pseudo distributed.
iii. Fully distributed.
6
b Use web based tools to monitor your Hadoop setup.
USING LINUX OPERATING SYSTEM
WEEK-3 Implementing the basic commands of LINUX Operating System –
a 24
File/Directory creation, deletion, update operations.
FILE MANAGEMENT IN HADOOP
Implement the following file management tasks in Hadoop:
i. Adding files and directories
WEEK-4 ii. Retrieving files
a iii. Deleting files 26
Hint: A typical Hadoop workflow creates data files (such as log files)
elsewhere and copies them into HDFS using one of the above
command line utilities.
MAPREDUCE PROGRAM 1
WEEK-5 Run a basic word count Map Reduce program to understand Map Reduce
28
Paradigm.
MAPREDUCE PROGRAM 2
Write a Map Reduce program that mines weather data. Hint: Weather sensors
WEEK-6 collecting data every hour at many locations across the globe gather a large
31
volume of log data, which is a good candidate for analysis with Map Reduce,
since it is semi structured and record-oriented.
MAPREDUCE PROGRAM 3
WEEK-7 Implement matrix multiplication with Hadoop Map Reduce 34
PIG LATIN LANGUAGE - PIG
WEEK-8
a Installation of PIG. 37
PIG COMMANDS
WEEK-9 a Write Pig Latin scripts sort, group, join, project, and filter your data. 39
7
WEEK-1
INSTALL VMWARE
1.1 OBJECTIVE:
To Install VMWare.
1.2 RESOURCES:
STEP 1. First of all, enter to the official site of VMware and download VMware Workstation
https://www.vmware.com/tryvmware/?p=workstation-w
8
Click on Next button and choose Typical option
STEP 4. By clicking “Next” buttons, to begin the installation, click on Install button at the end
STEP 5. This will install VMware Workstation software on your PC, After installation complete,
click on Finish button. Then restart your PC. Then open this software
9
6. In this step we try to create new “virtual machine”. Enter to File menu, then New-> Virtual
Machine
Then click Next button, and check your OS version. In this example, as we‟re going to setup
Oracle server on CentOS, we‟ll check Linux option and from “version” option we‟ll check Red Hat
10
Enterprise Linux 4
By clicking Next button, we‟ll give a name to our virtual machine, and give directory to create this new
virtual machine
11
Then select Use bridged networking option and click Next.
Then you‟ve to define size of hard disk by entering its size. I‟ll give 15 GB hard disk space and
please check Allocate all disk space now option
Here, you can delete Sound Adapter, Floppy and USB Controller by entering “Edit virtual machine
settings”. If you‟re going to setup Oracle Server, please make sure you‟ve increased your Memory
(RAM) to 1GB.
12
1.4 INPUT/OUTPUT
13
WEEK - 2
HADOOP MODES
2.1 OBJECTIVE:
2.2 RESOURCES:
a) STANDALONE MODE:
Installation of jdk 7
14
Create a ssh key for passwordless ssh configuration
15
<value>localhost:8021</value>
</property>
Format the name node
Command: start-dfs.sh
Start the task tracker and job tracker
Command: start-mapred.sh
To check if Hadoop started correctly
Command: jps
namenode
secondarynamenode
datanode
jobtracker
tasktracker
All the demons like namenodes and datanodes are runs on different machines. The data will replicate
according to the replication factor in client machines. The secondary namenode will store the mirror images
of namenode periodically. The namenode having the metadata where the blocks are stored and number of
replicas in the client machines. The slaves and master communicate each other periodically. The
configurations of multinode cluster are given below:
HDFS Namenode on UI
http://locahost:50070/
2.4 INPUT/OUTPUT:
ubuntu @localhost> jps
17
18
HDFS Jobtracker
http://locahost:50030/
HDFS Logs
http://locahost:50070/logs/
19
20
2.5 PRE LAB VIVA QUESTIONS:
21
WEEK- 3
USING LINUX OPERATING SYSTEM
3.1 OBJECTIVE:
1. Implementing the basic commands of LINUX Operating System – File/Directory creation,
deletion, update operations.
3.2 RESOURCES:
VMWare stack, 4 GB RAM, Hard Disk 80 GB.
3.3 PROGRAM LOGIC:
1.4 INPUT/OUTPUT:
Hint: A typical Hadoop workflow creates data files (such as log files) elsewhere and copies them into
HDFS using one of the above command line utilities.
4.2 RESOURCES:
VMWare stack, 4 GB RAM, Hard Disk 80 GB.
The Hadoop command get copies files from HDFS back to the local filesystem. To retrieve example.txt,
we can run the following command:
24
WEEK-5
MAPREDUCE PROGRAM 1
5.1 OBJECTIVE:
Run a basic word count Map Reduce program to understand Map Reduce Paradigm.
5.2 RESOURCES:
WordCount is a simple program which counts the number of occurrences of each word in a
given text input data set. WordCount fits very well with the MapReduce programming model
making it a great example to understand the Hadoop Map/Reduce programming style. Our
implementation consists of three main parts:
1. Mapper
2. Reducer
3. Driver
A Mapper overrides the ―map‖ function from the Class "org.apache.hadoop.mapreduce.Mapper" which
provides <key, value> pairs as the input. A Mapper implementation may output
<key,value> pairs using the provided Context .
Input value of the WordCount Map task will be a line of text from the input data file and the key
would be the line number <line_number, line_of_text> . Map task outputs <word, one> for each
word in the line of text.
Pseudo-code
output.collect(x,1);
25
Step-2. Write a Reducer
A Reducer collects the intermediate <key,value> output from multiple map tasks and assemble a single
result. Here, the WordCount program will sum up the occurrence of each word to pairs as
<word, occurrence>.
Pseudo-code
sum+=x;
final_output.collect(keyword, sum);
5.4 INPUT/OUTPUT:
26
5.6 LAB ASSIGNMENT:
Run a basic word count Map Reduce program to understand Map Reduce Paradigm.
27
WEEK-6
MAPREDUCE PROGRAM 2
6.1 OBJECTIVE:
Write a Map Reduce program that mines weather data. Hint: Weather sensors collecting data
every hour at many locations across the globe gather a large volume of log data, which is a
good candidate for analysis with Map Reduce, since it is semi structured and record-
oriented.
6.2 RESOURCES:
WordCount is a simple program which counts the number of occurrences of each word in a
given text input data set. WordCount fits very well with the MapReduce programming
model making it a great example to understand the Hadoop Map/Reduce programming style.
Our implementation consists of three main parts:
1. Mapper
2. Reducer
3. Main program
Pseudo-code
void Map (key, value){
for each max_temp x in value:
output.collect(x, 1);
}
void Map (key, value){
for each min_temp x in value:
output.collect(x, 1);
}
28
Step-2 Write a Reducer
A Reducer collects the intermediate <key,value> output from multiple map tasks and assemble a
single result. Here, the WordCount program will sum up the occurrence of each word to pairs as
<word, occurrence>.
Pseudo-code
void Reduce (max_temp, <list of value>){
for each x in <list of value>:
sum+=x;
final_output.collect(max_temp, sum);
}
void Reduce (min_temp, <list of value>){
for each x in <list of value>:
sum+=x;
final_output.collect(min_temp, sum);
}
3. Write Driver
The Driver program configures and run the MapReduce job. We use the main program to
perform basic configurations such as:
Job Name : name of this Job Executable (Jar)
Class: the main executable class. For here, WordCount.
Mapper Class: class which overrides the "map" function. For here, Map.
Reducer: class which override the "reduce" function. For here , Reduce.
Output Key: type of output key. For here, Text.Output
Value: type of output value. For here, IntWritable.
File Input Path
File Output Path
29
6.4 INPUT/OUTPUT:
Set of Weather Data over the years
30
WEEK-7
MAPREDUCE PROGRAM 3
7.1 OBJECTIVE:
Implement matrix multiplication with Hadoop Map Reduce.
7.2 RESOURCES:
VMWare, Web browser, 4 GB RAM, Hard Disk 80 GB.
The path of the directory for the output files for matrix C.
strategy = 1, 2, 3 or 4.
R = the number of reducers.
I = the number of rows in A and C.
K = the number of columns in A and rows in B.
J = the number of columns in B and C.
IB = the number of rows per A block and C block.
KB = the number of columns per A block and rows per B block.
JB = the number of columns per B block and C block.
In the pseudo-code for the individual strategies below, we have intentionally avoided
factoring common code for the purposes of clarity. Note that in all the strategies the memory
footprint of both the mappers and the reducers is flat at scale.
Note that the strategies all work reasonably well with both dense and sparse matrices. For sparse
matrices we do not emit zero elements. That said, the simple pseudo-code for multiplying the
individual blocks shown here is certainly not optimal for sparse matrices. As a learning exercise,
our focus here is on mastering the MapReduce complexities, not on optimizing the sequential
matrix multipliation algorithm for the individual blocks.
Steps
1. setup ()
2. var NIB = (I-1)/IB+1
3. var NKB = (K-1)/KB+1
4. var NJB = (J-1)/JB+1
5. map (key, value)
31
6. if from matrix A with key=(i,k) and value=a(i,k)
7. for 0 <= jb < NJB
8. emit (i/IB, k/KB, jb, 0), (i mod IB, k mod KB, a(i,k))
9. if from matrix B with key=(k,j) and value=b(k,j)
10. for 0 <= ib < NIB
emit (ib, k/KB, j/JB, 1), (k mod KB, j mod JB, b(k,j))
Intermediate keys (ib, kb, jb, m) sort in increasing order first by ib, then by kb, then by jb,
then by m. Note that m = 0 for A data and m = 1 for B data.
The partitioner maps intermediate key (ib, kb, jb, m) to a reducer r as follows:
11. r = ((ib*JB + jb)*KB + kb) mod R
12. These definitions for the sorting order and partitioner guarantee that each reducer
R[ib,kb,jb] receives the data it needs for blocks A[ib,kb] and B[kb,jb], with the data for
the A block immediately preceding the data for the B block.
13. var A = new matrix of dimension IBxKB
14. var B = new matrix of dimension KBxJB
15. var sib = -1
16. var skb = -1
Reduce (key, valueList)
17. if key is (ib, kb, jb, 0)
18. // Save the A block.
19. sib = ib
20. skb = kb
21. Zero matrix A
22. for each value = (i, k, v) in valueList A(i,k) = v
23. if key is (ib, kb, jb, 1)
24. if ib != sib or kb != skb return // A[ib,kb] must be zero!
25. // Build the B block.
26. Zero matrix B
27. for each value = (k, j, v) in valueList B(k,j) = v
28. // Multiply the blocks and emit the result.
29. ibase = ib*IB
30. jbase = jb*JB
31. for 0 <= i < row dimension of A
32. for 0 <= j < column dimension of B
33. sum = 0
34. for 0 <= k < column dimension of A = row dimension of B
a. sum += A(i,k)*B(k,j)
35. if sum != 0 emit (ibase+i, jbase+j), sum
Set of Data sets over different Clusters are taken as Rows and Columns
32
7.4 INPUT/OUTPUT:
33
WEEK-8
PIG LATIN LANGUAGE - PIG
8.1 OBJECTIVE:
1. Installation of PIG.
8.2 RESOURCES:
VMWare, Web browser, 4 GB RAM, Hard Disk 80 GB.
8.4 INPUT/OUTPUT:
Input as Website Click Count Data
34
8.5 PRE-LAB VIVA QUESTIONS:
1) What do you mean by a bag in Pig?
2) Differentiate between PigLatin and HiveQL
3) How will you merge the contents of two or more relations and divide a single relation
into two or more relations?
35
WEEK-9
PIG COMMANDS
9.1 OBJECTIVE:
Write Pig Latin scripts sort, group, join, project, and filter your data.
9.2 RESOURCES:
VMWare, Web browser, 4 GB RAM, Hard Disk 80 GB.
36
9.5 PRE-LAB VIVA QUESTIONS:
1. How will you merge the contents of two or more relations and divide a single relation
into two or more relations?
2. What is the usage of foreach operation in Pig scripts?
3. What does Flatten do in Pig?
37
WEEK-10
PIG LATIN MODES, PROGRAMS
10.1 OBJECTIVE:
a. Run the Pig Latin Scripts to find Word Count.
b. Run the Pig Latin Scripts to find a max temp for each and every year.
10.2 RESOURCES:
VMWare, Web Browser, 4 GB RAM, 80 GB Hard Disk.
Run the Pig Latin Scripts to find a max temp for each and every year
38
10.6 LAB ASSIGNMENT:
1. Analyzing average stock price from the stock data using Apache Pig
39
WEEK-11
HIVE
11.1 OBJECTIVE:
Installation of HIVE.
11.2 RESOURCES:
VMWare, Web Browser, 1GB RAM, Hard Disk 80 GB.
40
11.4 INPUT/OUTPUT:
41
WEEK-12
HIVE OPERATIONS
12.1 OBJECTIVE:
Use Hive to create, alter, and drop databases, tables, views, functions, and indexes.
12.2 RESOURCES:
VMWare, XAMPP Server, Web Browser, 1GB RAM, Hard Disk 80 GB.
42
INDEXES
CREATE INDEX index_name ON TABLE base_table_name (col_name, ...)
AS 'index.handler.class.name'
[WITH DEFERRED REBUILD]
[IDXPROPERTIES (property_name=property_value, ...)]
[IN TABLE index_table_name]
[PARTITIONED BY (col_name, ...)]
[
[ ROW FORMAT ...] STORED AS ...
| STORED BY ...
]
[LOCATION hdfs_path]
[TBLPROPERTIES (...)]
Creating Index
CREATE INDEX index_ip ON TABLE log_data(ip_address) AS
'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler' WITH DEFERRED
REBUILD;
Altering and Inserting Index
ALTER INDEX index_ip_address ON log_data REBUILD;
Storing Index Data in Metastore
SET
hive.index.compact.file=/home/administrator/Desktop/big/metastore_db/tmp/index_ipadd
ress_result;
SET
hive.input.format=org.apache.hadoop.hive.ql.index.compact.HiveCompactIndexInputFor
mat;
Dropping Index
DROP INDEX INDEX_NAME on TABLE_NAME;
43
12.4 INPUT/OUTPUT:
44
12.5 PRE-LAB VIVA QUESTIONS:
1. How many types of joins are there in Pig Latin with an examples?
2. Write the Hive command to create a table with four columns: First name, last name,
age, and income?
45