Midhun BIGDATA Curicullum
Midhun BIGDATA Curicullum
Module 02 - SQOOP
Sqoop Introduction
Sqoop Internal Process
Sqoop Explanation with Example
Sqoop with Eval
Sqoop with Split by
Handson1 - Eval,SplitBy,Basic Import from MySQL
Sqoop Import Properties
Sqoop Incremental Import
Handson2 - Sqoop Incremental Import
Sqoop Incremental last_Modified
Handson-3 Sqoop Incremental Append
Sqoop Job Creation - Basic
Sqoop Job creating Password file
Direct Mode
Sqoop Import using Shell sripting
Sqoop Handson Session -2
Sqoop validate Command
Sqoop Import into Hive table
Sqoop Import All Tables
Sqoop Import All Tables Exclude command
Sqoop Export Introduction
Sqoop Export Internal Process
Sqoop Export Incremental load
Sqoop Export properties
Sqoop Export Transcationality
Assignment-2
Interview Process Discussion Sqoop
Module 03 - HIVE
Hive Introduction
Why HQL
HQL VS SQL
Hive Architecture
Different Types of Hive metastore
Different ways of Accessing Hive
Hive Beeline Explanation
Different types of Execution Engines in Hive
Hive - Hadoop Integration
Hive - Tables - Managed and External Tables
Hive Internal tables Explanation
How to create the Internal tables
Hive Internal Table creation on top of Directory
Loading Data from a File to Hive table
Hive External tables Explanation
How to external Tables on dirtectory
Difference between Internal and External table
Handson - Internal And External Tables
Partitions Introduction
Static Partition - Load and Insert
Dynamic Partitions Insert
Handson - Static and Dynamic Partitions
Hive Sub partitions Explanation
Handson - Sub partitions
Bucketing in Hive Explanation
Bucketing on INTEGER column
Bucketing on String Column
Bucketing in Date Column
Handson- Bucketing
Bucketing and Partition on Same Table
Hive Query Optimization
Hive built In Functions
Views in Hive
Hive Sub quires
SCD Types Explanation in Hive
Implementation of SCD Type 1 in Hive
How to remove the duplicates in Hive table
Hive Serde Properties Explanation
Hive table creation on parquet
Hive Table creation on Avro
Hive table creation on XML files
Different types of Joins in Hive
Map side Join In Hive
Bucket Map Join in Hive
Sort Merge Bucket Join in Hibe
Handson - Joins in Hive
Hive UDF creation
Handson - Handle Incremental Load in Hive through Views
Hive Ranking functions
Concept of Vectorization
Choosing File format in Hive - Industry based
Hive MSCK command Explanation
Hive Advanced commands
ACID Properties In Hive
Handson - DML operations in Hive
ORC vs Parquet Vs AVRO
Interview Session - 1
Interview Session - 2
Assignment - 3
Assignment - 3 Solution
Assignment -4
Assignment -4 Solution
Project 1 - Sqoop Hive Data Process Pipeline Creation
Hbase
Introduction to Hbase
Types of NOSQL Databases
Characteristrics of NOSQL
CAP THEOREM
Why Column Based Storage is highly preferred than Row Based
RDBMS vs Hbase
Storage Hierarchy in HBASE
Hbase Architecture
TABLE design HBASE
What is column family in Hbase ?
Handson Session on HBASE commands
How to create the Hbase table
How to insert the data into Hbase Table
How to scan the data
How to enable the table
How to disable the table
Assignment-5
Assignment-5 Solution
Project 2 : Sqoop Hive Hbase Data processing Pipeline
Scala
Basics of Scala
Why Scala is called Functional programming
What is the use of Function programming
Difference between VAL and VAR
Data Types of Scala
Use of UNIT Data Type
Collection in Scala
List Collection
Set collection
Tuple Collection
Map Collection
Range Collection
Expressions in Scala
Statements in Scala
Scala Class Hierarchy
For Loop
If loop
Match Expression
Wild Card Pattern Matching
String Interpolation
Functions in Scala
Methods and Operations
Nested Functions
Variable Args Functions
Vector collection explanation
Recursive functions
Higher Order Functions Introduction
What is the use of Higher Order functions
Map Higher Order function
Filter Higher Order function
Foreach higher Order function
Reduceby Higher Order function
Currying In scala
Singleton Object on Scala
How to create a singeton object
Classes in Scala
Companion Object and Case Class
Main method in Scala
Factory Design Pattern
Traits in Scala
How to create traits in Projects
Options in Scala
Handling Nulls in Scala
Assignment - 6
Assignment - 6 Solution
SPARK
Spark Introduction
Why Spark?
Spark Ecosystem Components
Spark and mapReduce differences
Architecture of Spark
Different ways of process the data in Spark
Spark Core Introduction
What is SparkContext?
what is RDD and its importance?
what is DAG?
RDD Lineage
Concept of resilent
Lazy transformations
What is transformation in RDD
Examples of Transformations in RDD
What is actions in RDD ?
Examples of RDD Actions
Narrow and Wide Transformation
Setting Up Eclipse IDE
How to perform word count processing in Spark Core
Jar creationJar deployment
Spark Submit Introduction
Spark Submit Architecture explanation
Spark Submit - Stages in Spark
Different modes of Spark Submit
Spark Submit in Client mode
Spark Submit In cluster mode
Spark submit in Standalone mode
Spark Dynamic memory Allocation of resources
Difference between Group By Vs ReduceBy
Concept of Accumulators
Concepts of Broadcast varibales
How to Accumulators and broadcast variables acts as a Optimization techniques in Spark
Repartition
Coalesce
Difference between repartition and Coalesce - Real time scenerio
How to increase the parallelism in spark
Hands On Document for Spark Core
Spark Core HandsOn Session -1
Spark Core HandsOn Session -2
Concept of Map partition
Cache Concept In Detail
Units of Caching
Different memory Levels in Spark
Difference between cache vs persist
Concept of Serialization in Spark
Java serialization
Kyro Serialization
why Kyro Serialization is best for Spark?
Joins in Spark Core
Benefits of Repartitions
partitionBy vs bucketBy
saving file in various file format
Assignment - 7
Assignment - 7 Solution
Interview Preparation for Spark Core
Real time Code preparation for Spark Core in Eclipse using Business Logic
Module 11-KAFKA
Introduction to KAFKA
Why Kafka?
Kafka explanation with real time scenerio
Kafka Message Queue Components explanation
Topic,partition,Replication
What is Producer and Consumer?
Broker and its importance
Controller Broker explanation and its election
Use of Zookeeper
What is Offset ?
what is BootStrap Servers?
Installing One Node Kafka cluster locally
Introduction to KAFKA
Data storage in Brokers
Leader Copy in Kafka
Follower copy in Kafka
Consumer Groups
Data Serialization in Kafka
Assignment - 10
Assignment - 10 Solution
Project 4 : Kafka Spark Streaming Usecase
Module 12 - Cassandra
what is cassandra?
CAP THEOREM
Cassandra VS RDBMS
NoSQL DB Comparisions
Features of Cassandra
KeySpaces
Replica placement Strategy
Simple Strategy
Network Topology Strategy
CQL Table
Cassandra Handson Session