Kelly Hadoop Hyd May 2018
Kelly Hadoop Hyd May 2018
Flat No: 212, 2nd Floor, Annapurna Block, Aditya Enclave, Ameerpet, Hyd
[email protected] www.kellytechno.com Ph: 998 570 6789. Online: 001 973 780 6789.
Mr. GOPAL KRISHNA, Sr. Hadoop Technical Architect, BIGDATA Practice – CoE Lead
15+ Years Of Real Time IT Exp, 9+ Years On BIGDATA Projects Exp
CLOUDERA CCA 175 – Spark and Hadoop Certified Consultant
• HDFS Architecture - 5 Daemons of Hadoop
• NameNode and its functionality
• DataNode and its functionality
• JobTracker and its functionality
• TaskTrack and its functionality
• Secondary Name Node and its functionality.
• Accessing HDFS
• CLI (Command Line Interface) and HDFS Commands
• Java Based Approach
• Hadoop Archives
• Configuration files in Hadoop Installation and the Purpose
• How to & Where to Configure Hadoop Daemons in a Hadoop Cluster?
• Difference between Hadoop 1.X.X , Hadoop 2.X.X & 3.X.X version
• Name Node HA (High Availability in Hadoop 2.X.X)
• Importance of NFS in Hadoop-2.X
• Importance of Journal Nodes in Hadoop-2.X
MapReduce
• Input Split
InputSplit
Need Of Input Split in Map Reduce
InputSplit Size
InputSplit Size Vs Block Size
Flat No: 212, 2nd Floor, Annapurna Block, Aditya Enclave, Ameerpet, Hyd
[email protected] www.kellytechno.com Ph: 998 570 6789. Online: 001 973 780 6789.
Mr. GOPAL KRISHNA, Sr. Hadoop Technical Architect, BIGDATA Practice – CoE Lead
15+ Years Of Real Time IT Exp, 9+ Years On BIGDATA Projects Exp
CLOUDERA CCA 175 – Spark and Hadoop Certified Consultant
InputSplit Vs Mappers
• Driver Code
Importance of Driver Code in a Map Reduce program
How to Identify the Driver Code in Map Reduce program
Different sections of Driver code
• Mapper Code
Importance of Mapper Phase in Map Reduce
How to Write a Mapper Class?
Methods in Mapper Class
• Reducer Code
Importance of Reduce phase in Map Reduce
How to Write Reducer Class?
Methods in Reducer Class
• IDENTITY MAPPER & IDENTITY REDUCER
• Input Format’s in Map Reduce
TextInputFormat
KeyValueTextInputFormat
NLineInputFormat
DBInputFormat
SequenceFileInputFormat.
How to use the specific input format in Map Reduce
How to write Custom Input Format Class and Custom Record
Reader
• Output Format’s in Map Reduce
Flat No: 212, 2nd Floor, Annapurna Block, Aditya Enclave, Ameerpet, Hyd
[email protected] www.kellytechno.com Ph: 998 570 6789. Online: 001 973 780 6789.
Mr. GOPAL KRISHNA, Sr. Hadoop Technical Architect, BIGDATA Practice – CoE Lead
15+ Years Of Real Time IT Exp, 9+ Years On BIGDATA Projects Exp
CLOUDERA CCA 175 – Spark and Hadoop Certified Consultant
TextOutputFormat
KeyValueTextOutputFormat
NLineOutputFormat
DBOutputFormat
SequenceFileOutputFormat.
How to use the specific Output format in Map Reduce
How to write Custom Output Format Class and Custom Record
Writer
Apache PIG
• Introduction to Apache Pig
• Map Reduce Vs Apache Pig
• SQL Vs Apache Pig
• Different datat ypes in Pig
• Where to Use Map Reduce and PIG in REAL Time Hadoop Projects
• Modes Of Execution in Pig
Local Mode
Map Reduce OR Distributed Mode
• Execution Mechanism
Grunt Shell
Script
Embedded
• Transformations in Pig
• How to write a simple pig script
• Parameter substitution in PIG Scripts
• XML Processing through PIG
• JSON Processing through PIG
• Importance of DEFINE Keyword in PIG
• How to develop the Complex Pig Script
• Bags , Tuples and fields in PIG
• UDFs in Pig
Need of using UDFs in PIG
How to use UDFs
REGISTER Key word in PIG
• Techniques to improve the performance and efficiency of Pig Latin Programs
HIVE
• Hive Introduction
Flat No: 212, 2nd Floor, Annapurna Block, Aditya Enclave, Ameerpet, Hyd
[email protected] www.kellytechno.com Ph: 998 570 6789. Online: 001 973 780 6789.
Mr. GOPAL KRISHNA, Sr. Hadoop Technical Architect, BIGDATA Practice – CoE Lead
15+ Years Of Real Time IT Exp, 9+ Years On BIGDATA Projects Exp
CLOUDERA CCA 175 – Spark and Hadoop Certified Consultant
• Need of Apache HIVE in Hadoop
• When to choose MAP REDUCE , PIG & HIVE in REAL Time Project
• Hive Architecture
Driver
Compiler
Executor(Semantic Analyzer)
SQOOP
• Introduction to Sqoop.
• MySQL client and Server Installation
• How to connect to Relational Database using Sqoop
• Performance Implications in SQOOP Import and how to improve the
performance
• Performance Implications in SQOOP Export and how to improve the
performance
• Different Sqoop Commands
• Different flavors of Imports
• Export
• Hive-Imports
HBase
Flat No: 212, 2nd Floor, Annapurna Block, Aditya Enclave, Ameerpet, Hyd
[email protected] www.kellytechno.com Ph: 998 570 6789. Online: 001 973 780 6789.
Mr. GOPAL KRISHNA, Sr. Hadoop Technical Architect, BIGDATA Practice – CoE Lead
15+ Years Of Real Time IT Exp, 9+ Years On BIGDATA Projects Exp
CLOUDERA CCA 175 – Spark and Hadoop Certified Consultant
• HDFS Vs HBase
• HBase Vs RDBMS
• HBase Vs Cassandra VS Mongo DB & Real Time Use Cases on applicabiltiy
• Hbase usecases
• Hbase Data modeling Elements
• Column families
• Column Qualifier Name
• Row Key
• HBase Architecture
• Bulk Loading Operation with HBASE
• Importance of ImportTsv Utility in HBase
• Real Time case study on the usage of ImportTSV Utility of HBase
• Clients
• REST
• Thrift
• Java Based
• Avro
Flume
• Flume Introduction
• Flume Architecture
• Flume Master , Flume Collector and Flume Agent
• Flume Configurations
• Real Time Use Case using Apache Flume
• Sentimental Data Analytics with respect to Social Media Data with Flume &
Hive
Oozie
• Oozie Introduction
• Oozie Architectrure
• Oozie Configuration Files
• Oozie Job Submission
Workflow.xml
Coordinator.xml
Flat No: 212, 2nd Floor, Annapurna Block, Aditya Enclave, Ameerpet, Hyd
[email protected] www.kellytechno.com Ph: 998 570 6789. Online: 001 973 780 6789.
Mr. GOPAL KRISHNA, Sr. Hadoop Technical Architect, BIGDATA Practice – CoE Lead
15+ Years Of Real Time IT Exp, 9+ Years On BIGDATA Projects Exp
CLOUDERA CCA 175 – Spark and Hadoop Certified Consultant
job.coordinator.properties
Transit parameters in workflow.xml
Apache Cassandra
• Introduction to Cassandra
• Mongo DB Vs Cassandra
• Basic Operation using Cassandra
• Comparison among HBase , Mongo DB and Cassandra NO SQL DBs
Flat No: 212, 2nd Floor, Annapurna Block, Aditya Enclave, Ameerpet, Hyd
[email protected] www.kellytechno.com Ph: 998 570 6789. Online: 001 973 780 6789.
Mr. GOPAL KRISHNA, Sr. Hadoop Technical Architect, BIGDATA Practice – CoE Lead
15+ Years Of Real Time IT Exp, 9+ Years On BIGDATA Projects Exp
CLOUDERA CCA 175 – Spark and Hadoop Certified Consultant
Introduction to SCALA
• Why Scala
• Scala Vs Java
• Why Scala is a Hybrid Language
• Pre-Requisits for Scala Installation
SCALA Basics
• Scala Data types
• Scala Packages
• Runtime environment of Scala & Java
• Different IDE Support for Scala
• Control Structures
Interactive SCALA – SCALA Shell
• Scala REPL [ Real Evaluate Print Loop ]
• Writing Scala Scripts
• Compiling the Scala Programs
• Different IDEs for Scala
SCALA Type Less, Do More
• Var[variable] VS val[Value]
• Type Inference
• DataTyes in SCALA
• Type Casting in Scala
Conditional Statements in SCALA
Flat No: 212, 2nd Floor, Annapurna Block, Aditya Enclave, Ameerpet, Hyd
[email protected] www.kellytechno.com Ph: 998 570 6789. Online: 001 973 780 6789.
Mr. GOPAL KRISHNA, Sr. Hadoop Technical Architect, BIGDATA Practice – CoE Lead
15+ Years Of Real Time IT Exp, 9+ Years On BIGDATA Projects Exp
CLOUDERA CCA 175 – Spark and Hadoop Certified Consultant
• If expression
• If-else expression
• While Loop and Do…While Loop & difference between the two
• For loop , different forms of for loop in SCALA
• Pattern matching in SCALA & use of case and match keywords in SCALA
SCALA Collections
• List
• Set
• Map
SCALA Object Oriented Programming Introducton
SPARK
• Introduction to Spark
Spark SQL
• Introduction to Spark SQL
• The SQL Context
• Hive Vs Spark SQL
• Introduction to Data Frames [ DFs ]
• Examples on Spark SQL
Different File Formats Processing through Spark SQL
• CSV
• JSON
• PARQUET
• ORC
• TEXT
Flat No: 212, 2nd Floor, Annapurna Block, Aditya Enclave, Ameerpet, Hyd
[email protected] www.kellytechno.com Ph: 998 570 6789. Online: 001 973 780 6789.
Mr. GOPAL KRISHNA, Sr. Hadoop Technical Architect, BIGDATA Practice – CoE Lead
15+ Years Of Real Time IT Exp, 9+ Years On BIGDATA Projects Exp
CLOUDERA CCA 175 – Spark and Hadoop Certified Consultant
Spark SQL Integrations
• Spark – Hive Integration and Real Time use cases on the same
• Spark – RDBMS Integration and Real Time use cases on the same
• Spark – NO SQL Integration Introduction and Importance
Flat No: 212, 2nd Floor, Annapurna Block, Aditya Enclave, Ameerpet, Hyd
[email protected] www.kellytechno.com Ph: 998 570 6789. Online: 001 973 780 6789.