Professional Documents
Culture Documents
Hadoop Interview Question
Hadoop Interview Question
Hadoop Certification
(Developer, Administrator
HBase & Data Science)
CCD-410, CCA-410 and
CCB-400 and DS-200
Note: There is 50% talent gap in BigData domain, get Hadoop certification with the
Hadoop Learning Resources Hadoop Exam Simulator.
©Hadoop Learning Resources 3
(Guarantee you love it...the way trainer is teaching... just watch sample two modules..)
First two modules are free (Best Quality is Promise, watch it right now) Best Hadoop Training Quality Guaranteed among
all Vendors. Believe in us you will say wow...it’s Great. after watching first two modules
Module 1 : Introduction to BigData, Hadoop (HDFS and MapReduce) : Available (Length 35 Minutes)
1. BigData Inroduction
2. Hadoop Introduction
3. HDFS Introduction
4. MapReduce Introduction
1. HDFS Design
2. Fundamental of HDFS (Blocks, NameNode, DataNode, Secondary Name Node)
3. Rack Awareness
4. Read/Write from HDFS
5. HDFS Federation and High Availability
6. Parallel Copying using DistCp
7. HDFS Command Line Interface
Video URL : http://www.youtube.com/watch?v=PK6Im7tBWow
Module 5 : MapReduce-2 (YARN : Yet Another Resource Negotiator) : Available (Length 52 Minutes)
Module 6 : Advanced Topic for MapReduce (Performance and Optimization) : Available (Length 58 Minutes)
1. Job Sceduling
2. In Depth Shuffle and Sorting
3. Speculative Execution
4. Output Committers
5. JVM Reuse in MR1
6. Configuration and Performance Tuning
1. MapReduce Joining
- Reduce Side Join
- MapSide Join
- Semi Join
2. MapReduce Job Chaining
- MapReduce Sequence Chaining
- MapReduce Complex Chaining
Module 10: MapReduce DataTypes and Formats : Available : Private (Length 77 Minutes)
1.Serialization In Hadoop
2. Hadoop Writable and Comparable
3. Hadoop RawComparator and Custom Writable
4. MapReduce Types and Formats
5. Understand Difference Between Block and InputSplit
6. Role of RecordReader
7. FileInputFormat
8. ComineFileInputFormat and Processing whole file Single Mapper
9. Each input File as a record
10. Text/KeyValue/NLine InputFormat
11. BinaryInput processing
12. MultipleInputs Format
13. DatabaseInput and Output
14. Text/Biinary/Multiple/Lazy OutputFormat MapReduce Types
1. What is Pig ?
2. Introduction to Pig Data Flow Engine
3. Pig and MapReduce in Detail
4. When should Pig Used ?
5. Pig and Hadoop Cluster
6. Pig Interpreter and MapReduce
©Hadoop Learning Resources 6
7. Pig Relations and Data Types
8. PigLatin Example in Detail
9. Debugging and Generating Example in Apache Pig
1. What is Hive ?
2. Architecture of Hive
3. Hive Services
4. Hive Clients
5. how Hive Differs from Traditional RDBMS
6. Introduction to HiveQL
7. Data Types and File Formats in Hive
8. File Encoding
9. Common problems while working with Hive
Video URL : Watch Private Video
Module 14 : Hands On : Single Node Hadoop Cluster Set Up In Amazon Cloud : Available (Length 60 Minutes Hands
On Practice Session)
1. � How to create instance on Amazon EC2
2. � How to connect that Instance Using putty
3. � Installing Hadoop framework on this instance
4. � Run sample wordcount example which come with Hadoop framework.
In 30 minutes you can create Hadoop Single Node Cluster in Amazon cloud, does it interest you ?
Module 15 : Hands On : Implementation of NGram algorithm : Available (Length 48 Minutes Hands On Practice
Session)
1. Understand the NGram concept using (Google Books NGram )
2. Step by Step Process creating and Configuring eclipse for writing MapReduce Code
3. Deploying the NGram application in Hadoop Installed in Amazon EC2
4. Analyzing the Result by Running NGram application (UniGram, BiGram, TriGram etc.)
©Hadoop Learning Resources 7
Module 16 : Hands On : Hadoop MultiNode Cluster Setup and Running a BigData Example : Available (Length 70
Minutes)
1. Hadoop MultiNode Cluster
2. Setup Three Node Hadoop cluster
3. Running NGram Application on cluster
4. Analyze the cluster using
-- NameNode UI (Multiple Blocks and effect of Replication Factor)
-- JobTracker UI (Multiple MapTask running on different Nodes)
5. SettingUp Replication Factor
Reduce Task : And the above output will be the input for the reducetasks, produces
A map task.
A reduced task.
Job configuration
The Hadoop job client then submits the job (jar/executable etc.) and configuration
to the JobTracker which then assumes the responsibility of distributing the
software/configuration to the slaves, scheduling tasks and monitoring them,
providing status and diagnostic information to the job-client.
7. Explain how input and output data format of the Hadoop framework?
©Hadoop Learning Resources 10
Ans: The MapReduce framework operates exclusively on pairs, that is, the
framework views the input to the job as a set of pairs and produces a set of pairs
as the output of the job, conceivably of different types. See the flow mentioned below
(input) -> map -> -> combine/sorting -> -> reduce -> (output)
Note: There is 50% talent gap in BigData domain, get Hadoop certification with the
Hadoop Learning Resources Hadoop Exam Simulator.
Ans: The Mapper itself is instantiated in the running job, and will be passed a
MapContext object which it can use to configure itself.
21. Which object can be used to get the progress of a particular job ?
Ans: Context
22. What is next step after Mapper or MapTask?
Ans : The output of the Mapper are sorted and Partitions will be created for the
output. Number of partition depends on the number of reducer.
intermediate outputs, which helps to cut down the amount of data transferred
from the Mapper to the
Reducer.
25. How many maps are there in a particular Job?
Ans: The number of maps is usually driven by the total size of the inputs, that is,
the total number of blocks of the input files.
Generally it is around 10-100 maps per-node. Task setup takes awhile, so it is
best if the maps take at least a minute to execute.
Suppose, if you expect 10TB of input data and have a blocksize of 128MB, you'll
end up with
82,000 maps, to control the number of block you can use the
mapreduce.job.maps parameter (which only provides a hint to the framework).
Ultimately, the number of tasks is controlled by the number of splits returned by
the InputFormat.getSplits() method (which you can override).
26. What is the Reducer used for?
Ans: Reducer reduces a set of intermediate values which share a key to a
(usually smaller) set of values.
The number of reduces for the job is set by the user via
Job.setNumReduceTasks(int).
Ans: The framework groups Reducer inputs by keys (since different mappers may
have output the same key) in this stage. The shuffle and sort phases occur simultaneously;
while map-outputs are being fetched they are merged (It is similar to merge-sort).
The TaskTracker nodes are monitored. If they do not submit heartbeat signals
often enough, they are deemed to have failed and the work is scheduled on a
different TaskTracker.
A TaskTracker will notify the JobTracker when a task fails. The JobTracker
decides what to do then: it may resubmit the job elsewhere, it may mark that
specific record as something to avoid, and it may may even blacklist the
TaskTracker as unreliable.
When the work is completed, the JobTracker updates its status.
Client applications can poll the JobTracker for information.
The Tasktracker monitors these task instances, capturing the output and exit
codes. When the Task instances finish, successfully or not, the task tracker
notifies the JobTracker.
The TaskTrackers also send out heartbeat messages to the JobTracker, usually
every few minutes, to reassure the JobTracker that it is still alive. These
messages also inform the JobTracker of the number of available slots, so the
JobTracker can stay up to date with where in the cluster work can be delegated .
Ans: One or Multiple instances of Task Instance can run on each slave node.
Each task instance is run as a separate JVM process. The number of Task
instances can be controlled by configuration. Typically a high end machine is
configured to run more task instances.
©Hadoop Learning Resources 19
o In HDFS Data Blocks are distributed across local drives of all machines in a
cluster. Whereas in NAS data is stored
on dedicated hardware.
o HDFS is designed to work with MapReduce System, since computation is
moved to data. NAS is not suitable for MapReduce since data is stored
separately from the computations.
o HDFS runs on a cluster of machines and provides redundancy using
replication protocol. Whereas NAS is provided by a single machine
therefore does not provide data redundancy.
46. How does a NameNode handle the failure of the data nodes?
Ans: HDFS has master/slave architecture. An HDFS cluster consists of a single
NameNode, a master server that manages the file system namespace and regulates
access to files by clients.
In addition, there are a number of DataNodes, usually one per node in the cluster,
which manage storage attached to the nodes that they run on.
The NameNode and DataNode are pieces of software designed to run on commodity
machines.
NameNode periodically receives a Heartbeat and a Block report from each of the
DataNodes in the cluster. Receipt of a Heartbeat implies that the DataNode is
functioning properly. A Blockreport contains a list of all blocks on a DataNode. When
NameNode notices that it has not received a heartbeat message from a data node
after a certain amount of time, the data node is marked as dead. Since blocks will be
under replicated the system begins replicating the blocks that were stored on the
dead DataNode. The NameNode Orchestrates the replication of data blocks from
©Hadoop Learning Resources 20
one DataNode to another. The replication data transfer happens directly between
DataNode and the data never passes through the NameNode.
50. What is the Hadoop MapReduce API contract for a key and value Class?
Ans:
◦The Key must implement the org.apache.hadoop.io.WritableComparable
interface.
◦The value must implement the org.apache.hadoop.io.Writable interface.
It has many similarities with existing distributed file systems. However, the
differences from other distributed file systems are significant.
◦HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware.
◦HDFS provides high throughput access to application data and is suitable for
applications that have large data sets.
◦HDFS is designed to support very large files. Applications that are compatible with
HDFS are those that deal with large data sets. These applications write their data
only once but they read it one or more times and require these reads to be satisfied
at streaming speeds. HDFS supports write-once-read-many semantics on files.
55. What is HDFS Block size? How is it different from traditional file system
block size?
In HDFS data is split into blocks and distributed across multiple nodes in the cluster.
Each block is typically 64Mb or 128Mb in size.
Each block is replicated multiple times. Default is to replicate each block three times.
Replicas are stored on different nodes. HDFS utilizes the local file system to store
each HDFS block as a separate file. HDFS Block size can not be compared with the
traditional file system block size.
Note: There is 50% talent gap in BigData domain, get Hadoop certification with the
Hadoop Learning Resources Hadoop Exam Simulator.
Contact Us
Email : [email protected]
[email protected]
Phone : 022-42669636
Mobile : +91-8879712614
HadoopExam Learning Resources
B-902 Shah Trade Center
Malad-E Mumbai 400097
©Hadoop Learning Resources 25