Data Lake for Enterprises
By Pankaj Misra and Tomcy John
()
About this ebook
- Build a full-fledged data lake for your organization with popular big data technologies using the Lambda architecture as the base
- Delve into the big data technologies required to meet modern day business strategies
- A highly practical guide to implementing enterprise data lakes with lots of examples and real-world use-cases
Java developers and architects who would like to implement a data lake for their enterprise will find this book useful. If you want to get hands-on experience with the Lambda Architecture and big data technologies by implementing a practical solution using these technologies, this book will also help you.
Pankaj Misra
The Chronicles of Devi - Cosmos & Mahisha is my second book. The good response to my first book The Himalayan Revelation has encouraged me to continue penning my effervescent ruminations. I am an engineer by education and a technologist by profession and more than anything else I love travelling and exploring new places.During my two decade active corporate career and travels into the lap of nature and history, I have came across many people, places and the tales associated with them which have left me enriched by some told and some often untold stories.The Himalayas of Ladakh and Hampi are couple of those places which have left an everlasting impression on me. Their enigmatic environs hide many tales inside them, some of which have been passed from one generation to another. The Himalayan Revelation is my attempt to weave in some of those tales in a modern context. Though it is purely a work of fiction but the story is developed using facts gleamed from history and modern times.In The Chronicles of Devi, I have attempted to tell a story which has captivated many minds for generations. Our ancient literature is full of enchanting stories. But most of these stories are hidden in the beautiful intricacies of Sanskrit in Vedas, Purans, and Shlokas. I have picked one such gem and have narrated this story in my own modernistic interpretation while keeping its essence intact- it is a story of a single woman and her lion fighting the forces of evil to preserve the balance of nature.I hope you like both the books.Pankaj
Read more from Pankaj Misra
The Himalayan Revelation Rating: 0 out of 5 stars0 ratings
Related to Data Lake for Enterprises
Related ebooks
Data Lake Architecture Complete Self-Assessment Guide Rating: 0 out of 5 stars0 ratingsData Lake Development with Big Data Rating: 0 out of 5 stars0 ratingsArchitecting Big Data & Analytics Solutions - Integrated with IoT & Cloud Rating: 5 out of 5 stars5/5Big Data for Enterprise Architects Rating: 5 out of 5 stars5/5Data Engineering on Azure Rating: 0 out of 5 stars0 ratingsBuilding the Data Warehouse Rating: 5 out of 5 stars5/5Hadoop Real-World Solutions Cookbook - Second Edition Rating: 0 out of 5 stars0 ratingsData Pipelines with Apache Airflow Rating: 0 out of 5 stars0 ratingsDesigning Cloud Data Platforms Rating: 0 out of 5 stars0 ratingsDatabricks A Complete Guide - 2021 Edition Rating: 0 out of 5 stars0 ratingsData Architecture Complete Self-Assessment Guide Rating: 0 out of 5 stars0 ratingsData warehouse Complete Self-Assessment Guide Rating: 4 out of 5 stars4/5The Data Model Resource Book: Volume 3: Universal Patterns for Data Modeling Rating: 0 out of 5 stars0 ratingsIntegration Architecture Rating: 5 out of 5 stars5/5Learning PySpark Rating: 0 out of 5 stars0 ratingsData Analytics with Google Cloud Platform: Build Real Time Data Analytics on Google Cloud Platform Rating: 0 out of 5 stars0 ratingsAzure Storage, Streaming, and Batch Analytics: A guide for data engineers Rating: 0 out of 5 stars0 ratingsData Lake Architecture Strategy A Complete Guide - 2020 Edition Rating: 0 out of 5 stars0 ratingsMDM for Customer Data: Optimizing Customer Centric Management of Your Business Rating: 0 out of 5 stars0 ratingsOfficial Google Cloud Certified Professional Data Engineer Study Guide Rating: 5 out of 5 stars5/5
Enterprise Applications For You
Excel 2019 For Dummies Rating: 3 out of 5 stars3/5QuickBooks 2024 All-in-One For Dummies Rating: 0 out of 5 stars0 ratingsBitcoin For Dummies Rating: 4 out of 5 stars4/5Creating Online Courses with ChatGPT | A Step-by-Step Guide with Prompt Templates Rating: 4 out of 5 stars4/5Excel : The Ultimate Comprehensive Step-By-Step Guide to the Basics of Excel Programming: 1 Rating: 5 out of 5 stars5/5QuickBooks 2023 All-in-One For Dummies Rating: 0 out of 5 stars0 ratingsEnterprise AI For Dummies Rating: 3 out of 5 stars3/5Excel Formulas That Automate Tasks You No Longer Have Time For Rating: 5 out of 5 stars5/5Scrivener For Dummies Rating: 4 out of 5 stars4/5Excel 101: A Beginner's & Intermediate's Guide for Mastering the Quintessence of Microsoft Excel (2010-2019 & 365) in no time! Rating: 0 out of 5 stars0 ratingsNotion for Beginners: Notion for Work, Play, and Productivity Rating: 4 out of 5 stars4/5Excel 2016 For Dummies Rating: 4 out of 5 stars4/550 Useful Excel Functions: Excel Essentials, #3 Rating: 5 out of 5 stars5/5Excel Gurus Gone Wild: Do the IMPOSSIBLE with Microsoft Excel Rating: 4 out of 5 stars4/5Access 2019 For Dummies Rating: 0 out of 5 stars0 ratingsExcel Formulas and Functions 2020: Excel Academy, #1 Rating: 4 out of 5 stars4/5SharePoint 2016 For Dummies Rating: 5 out of 5 stars5/5Managing Humans: Biting and Humorous Tales of a Software Engineering Manager Rating: 4 out of 5 stars4/5Excel All-in-One For Dummies Rating: 0 out of 5 stars0 ratingsExcel Workbook For Dummies Rating: 4 out of 5 stars4/5Excel for the Math Classroom Rating: 5 out of 5 stars5/5Microsoft Excel 365 Bible Rating: 0 out of 5 stars0 ratingsQuickBooks Online For Dummies Rating: 0 out of 5 stars0 ratingsChatGPT Ultimate User Guide - How to Make Money Online Faster and More Precise Using AI Technology Rating: 1 out of 5 stars1/5
Reviews for Data Lake for Enterprises
0 ratings0 reviews
Book preview
Data Lake for Enterprises - Pankaj Misra
Title Page
Data Lake for Enterprises
Leveraging Lambda Architecture for building Enterprise Data Lake
Tomcy John
Pankaj Misra
BIRMINGHAM - MUMBAI
Copyright
Data Lake for Enterprises
Copyright © 2017 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: May 2017
Production reference: 1300517
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham
B3 2PB, UK.
ISBN 978-1-78728-134-9
www.packtpub.com
Credits
Foreword
As organizations have evolved over the last 40 to 50 years, they have slowly but steadily found ways and means to improve their operations by adding IT/software systems across their operating areas. It would not be surprising today to see more than 250+ applications in each of our Fortune 200 companies. This has also slowly caused another creeping problem as we evolve from our level of maturity to another; silos of systems that don’t interface well to each other.
As enterprises move from local optimization to enterprise optimization they have been leveraging some of the emerging technologies like Big Data systems to find ways and means by which they could bring data together from their disparate IT systems and fuse them together to find better means of driving efficiency and effectiveness improvement that could go a long way in helping enterprises save money.
Tomcy and Pankaj, with their vast experience in different functional and technical domains, have been working on finding better ways to fuse information from variety of applications within the organization. They have lived through the challenging journey of finding a ways to bring out changes (technological & cultural).
This book has been put together from the perspective of software engineers, architects and managers; so it’s very practical in nature as both of them have lived through various enterprise grade implementation that adds value to the enterprise.
Using future proof patterns and contemporary technology concepts like Data Lake help enterprises prepare themselves well for the future, but even more given them the ability to look across data that they have across different organizational silos and derive wisdom that’s typically lost in the blind spots.
Thomas Benjamin
CTO, GE Aviation Digital.
About the Authors
Tomcy John lives in Dubai (United Arab Emirates), hailing from Kerala (India), and is an enterprise Java specialist with a degree in engineering (B Tech) and over 14 years of experience in several industries. He's currently working as principal architect at Emirates Group IT, in their core architecture team. Prior to this, he worked with Oracle Corporation and Ernst & Young. His main specialization is in building enterprise-grade applications and he acts as chief mentor and evangelist to facilitate incorporating new technologies as corporate standards in the organization. Outside of his work, Tomcy works very closely with young developers and engineers as mentors and speaks at various forums as a technical evangelist on many topics ranging from web and middleware all the way to various persistence stores. He writes on various topics in his blog and www.javacodebook.com.
First and foremost, I would like to thank my savior and lord, Jesus Christ, for giving me strength and courage to pursue this project. It was a dream come true.
I would like to dedicate this book to my father (Appachan), Late C.O.John, and my dearest mom (Ammachi), Leela John, for helping me reach where I am today. I would also like to take this opportunity to thank my dearest wife, Serene and our two lovely children, Neil (son) and Anaya (daughter), for all their support throughout this project and also for allowing me to pursue my dream and tolerating not being with them after my busy day job.
It was my privilege working with my co-author, Pankaj. I take this opportunity to thank him for supporting me, when I first offloaded my dream of writing this book topic and then staying with me at all stages in completing this book. It wouldn't be possible to reach this stage in my career without mentors at various stages of my career. I would like to thank Thomas Benjamin (CTO, GE Aviation Digital), Rajesh R.V (chief architect, Emirates Group IT) and Martin Campbell (chief architect) for supporting me at various stages, with words of encouragement and wealth of knowledge.
Pankaj Misra has been a technology evangelist, holding a bachelor’s degree in engineering, with over 16 years of experience across multiple business domains and technologies. He has been working with Emirates Group IT since 2015, and has worked with various other organizations in the past. He specializes in architecting and building multi-stack solutions and implementations. He has also been a speaker at technology forums in India and has built products with scale-out architecture that support high-volume, near-real-time data processing and near-real-time analytics.
This book has been a great opportunity for me and would always be an exceptional example of collaboration and knowledge sharing with my co-author Tomcy. I am extremely thankful to him for entrusting me with this responsibility and standing by me at all times. I would like to dedicate this book to my father B. Misra and my mother Geeta Misra who have always been one of the most special people to me. I am extremely grateful to my wife Priti and my kids, daughter Eva and son Siddhant, for their understanding, support and helping me out in every possible way to complete the book.
This book is a medium to give back the knowledge that I have gained by working with many of the amazing people throughout the years. I would like to thank Rajesh R.V. (chief Architect, Emirates Group IT) and Thomas Benjamin (CTO, GE Aviation) for always motivating, helping & supporting us.
About the Reviewers
Wei Di is currently a staff member in a business analytics data mining team. As a data scientist, she is passionate about creating smart and scalable analytics and data mining solutions that can impact millions of individuals and empower successful business.
Her interests also cover wide areas, including artificial intelligence, machine learning, and computer vision. She was previously associated with the eBay human language technology team and eBay research labs, with focus on image understanding for large-scale application and joint learning from both visual and text information. Prior to that, she was with Ancestry.com, working on large-scale data mining and machine learning models in the areas of record linkage, search relevance, and ranking. She received her PhD from Purdue University in 2011 with focus on data mining and image classification.
Vivek Mishra is an IT professional with more than 9 years of experience in various technologies like Java, J2ee, Hibernate, SCA4J, Mule, Spring, Cassandra, HBase, MongoDB, REDIS, Hive, Hadoop. He has been a contributor to open source software such as Apache Cassandra and lead committer for Kundera(a JPA 2.0-compliant object-datastore mapping library for NoSQL Datastores such as Cassandra, HBase, MongoDB, and REDIS).
Vivek, in his previous experience, has enjoyed long-lasting partnerships with the most recognizable names in SCM, banking and finance industries, employing industry-standard, full-software life cycle methodologies such as Agile and SCRUM. He is currently employed with Impetus Infotech.
He has undertaken speaking engagements in cloud camp and Nasscom big data seminars and is an active blogger at mevivs.wordpress.com.
Rubén Oliva Ramos is a computer systems engineer with a master's degree in computer and electronic systems engineering, teleinformatics, and networking specialization from University of Salle Bajio in Leon, Guanajuato, Mexico. He has more than 5 years of experience in developing web applications to control and monitor devices connected with Arduino and Raspberry Pi using web frameworks and cloud services to build Internet of Things applications.
He is a mechatronics teacher at the University of Salle Bajio and teaches students of master's in design and engineering of mechatronics Systems. He also works at Centro de Bachillerato Tecnologico Industrial 225 in Leon, Guanajuato, Mexico, teaching the following: electronics, robotics and control, automation, and microcontrollers at Mechatronics Technician Career. He has worked on consultant and developer projects in areas such as monitoring systems and datalogger data using technologies such as Android, iOS, Windows Phone, Visual Studio .NET, HTML5, PHP, CSS, Ajax, JavaScript, Angular, ASP .NET databases (SQlite, mongoDB, and MySQL), and web servers (Node.js and IIS). Ruben has done hardware programming on Arduino, Raspberry Pi, Ethernet Shield, GPS and GSM/GPRS, ESP8266, and control and monitor systems for data acquisition and programming.
He has written the book titled Internet of Things Programming with JavaScript, Packt.
His current job involves monitoring, controlling, and acquisition of data with Arduino and Visual Basic .NET for Alfaomega Editor Group.
I want to thank God for helping me reviewing this book, to my wife, Mayte, and my sons, Ruben and Dario, for their support, to my parents, my brother and sister whom I love and to all my beautiful family.
www.PacktPub.com
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.comand as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details.
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.
https://www.packtpub.com/mapt
Get the most in-demand software skills with Mapt. Mapt gives you full access to all Packt books and video courses, as well as industry-leading tools to help you plan your personal development and advance your career.
Why subscribe?
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Customer Feedback
Thanks for purchasing this Packt book. At Packt, quality is at the heart of our editorial process. To help us improve, please leave us an honest review on this book's Amazon page at https://www.amazon.com/dp/1787281345.
If you'd like to join our team of regular reviewers, you can e-mail us at [email protected]. We award our regular reviewers with free eBooks and videos in exchange for their valuable feedback. Help us be relentless in improving our products!
Table of Contents
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Downloading the example code
Errata
Piracy
Questions
Introduction to Data
Exploring data
What is Enterprise Data?
Enterprise Data Management
Big data concepts
Big data and 4Vs
Relevance of data
Quality of data
Where does this data live in an enterprise?
Intranet (within enterprise)
Internet (external to enterprise)
Business applications hosted in cloud
Third-party cloud solutions
Social data (structured and unstructured)
Data stores or persistent stores (RDBMS or NoSQL)
Traditional data warehouse
File stores
Enterprise's current state
Enterprise digital transformation
Enterprises embarking on this journey
Some examples
Data lake use case enlightenment
Summary
Comprehensive Concepts of a Data Lake
What is a Data Lake?
Relevance to enterprises
How does a Data Lake help enterprises?
Data Lake benefits
How Data Lake works?
Differences between Data Lake and Data Warehouse
Approaches to building a Data Lake
Lambda Architecture-driven Data Lake
Data ingestion layer - ingest for processing and storage
Batch layer - batch processing of ingested data
Speed layer - near real time data processing
Data storage layer - store all data
Serving layer - data delivery and exports
Data acquisition layer - get data from source systems
Messaging Layer - guaranteed data delivery
Exploring the Data Ingestion Layer
Exploring the Lambda layer
Batch layer
Speed layer
Serving layer
Data push
Data pull
Data storage layer
Batch process layer
Speed layer
Serving layer
Relational data stores
Distributed data stores
Summary
Lambda Architecture as a Pattern for Data Lake
What is Lambda Architecture?
History of Lambda Architecture
Principles of Lambda Architecture
Fault-tolerant principle
Immutable Data principle
Re-computation principle
Components of a Lambda Architecture
Batch layer
Speed layer
CAP Theorem
Eventual consistency
Serving layer
Complete working of a Lambda Architecture
Advantages of Lambda Architecture
Disadvantages of Lambda Architectures
Technology overview for Lambda Architecture
Applied lambda
Enterprise-level log analysis
Capturing and analyzing sensor data
Real-time mailing platform statistics
Real-time sports analysis
Recommendation engines
Analyzing security threats
Multi-channel consumer behaviour
Working examples of Lambda Architecture
Kappa architecture
Summary
Applied Lambda for Data Lake
Knowing Hadoop distributions
Selection factors for a big data stack for enterprises
Technical capabilities
Ease of deployment and maintenance
Integration readiness
Batch layer for data processing
The NameNode server
The secondary NameNode Server
Yet Another Resource Negotiator (YARN)
Data storage nodes (DataNode)
Speed layer
Flume for data acquisition
Source for event sourcing
Interceptors for event interception
Channels for event flow
Sink as an event destination
Spark Streaming
DStreams
Data Frames
Checkpointing
Apache Flink
Serving layer
Data repository layer
Relational databases
Big data tables/views
Data services with data indexes
NoSQL databases
Data access layer
Data exports
Data publishing
Summary
Data Acquisition of Batch Data using Apache Sqoop
Context in data lake - data acquisition
Data acquisition layer
Data acquisition of batch data - technology mapping
Why Apache Sqoop
History of Sqoop
Advantages of Sqoop
Disadvantages of Sqoop
Workings of Sqoop
Sqoop 2 architecture
Sqoop 1 versus Sqoop 2
Ease of use
Ease of extension
Security
When to use Sqoop 1 and Sqoop 2
Functioning of Sqoop
Data import using Sqoop
Data export using Sqoop
Sqoop connectors
Types of Sqoop connectors
Sqoop support for HDFS
Sqoop working example
Installation and Configuration
Step 1 - Installing and verifying Java
Step 2 - Installing and verifying Hadoop
Step 3 - Installing and verifying Hue
Step 4 - Installing and verifying Sqoop
Step 5 - Installing and verifying PostgreSQL (RDBMS)
Step 6 - Installing and verifying HBase (NoSQL)
Configure data source (ingestion)
Sqoop configuration (database drivers)
Configuring HDFS as destination
Sqoop Import
Import complete database
Import selected tables
Import selected columns from a table
Import into HBase
Sqoop Export
Sqoop Job
Job command
Create job
List Job
Run Job
Create Job
Sqoop 2
Sqoop in purview of SCV use case
When to use Sqoop
When not to use Sqoop
Real-time Sqooping: a possibility?
Other options
Native big data connectors
Talend
Pentaho's Kettle (PDI - Pentaho Data Integration)
Summary
Data Acquisition of Stream Data using Apache Flume
Context in Data Lake: data acquisition
What is Stream Data?
Batch and stream data
Data acquisition of stream data - technology mapping
What is Flume?
Sqoop and Flume
Why Flume?
History of Flume
Advantages of Flume
Disadvantages of Flume
Flume architecture principles
The Flume Architecture
Distributed pipeline - Flume architecture
Fan Out - Flume architecture
Fan In - Flume architecture
Three tier design - Flume architecture
Advanced Flume architecture
Flume reliability level
Flume event - Stream Data
Flume agent
Flume agent configurations
Flume source
Custom Source
Flume Channel
Custom channel
Flume sink
Custom sink
Flume configuration
Flume transaction management
Other flume components
Channel processor
Interceptor
Channel Selector
Sink Groups
Sink Processor
Event Serializers
Context Routing
Flume working example
Installation and Configuration
Step 1: Installing and verifying Flume
Step 2: Configuring Flume
Step 3: Start Flume
Flume in purview of SCV use case
Kafka Installation
Example 1 - RDBMS to Kafka
Example 2: Spool messages to Kafka
Example 3: Interceptors
Example 4 - Memory channel, file channel, and Kafka channel
When to use Flume
When not to use Flume
Other options
Apache Flink
Apache NiFi
Summary
Messaging Layer using Apache Kafka
Context in Data Lake- messaging layer
Messaging layer
Messaging layer- technology mapping
What is Apache Kafka?
Why Apache Kafka
History of Kafka
Advantages of Kafka
Disadvantages of Kafka
Kafka architecture
Core architecture principles of Kafka
Data stream life cycle
Working of Kafka
Kafka message
Kafka producer
Persistence of data in Kafka using topics
Partitions- Kafka topic division
Kafka message broker
Kafka consumer
Consumer groups
Other Kafka components
Zookeeper
MirrorMaker
Kafka programming interface
Kafka core API's
Kafka REST interface
Producer and consumer reliability
Kafka security
Kafka as message-oriented middleware
Scale-out architecture with Kafka
Kafka connect
Kafka working example
Installation
Producer - putting messages into Kafka
Kafka Connect
Consumer - getting messages from Kafka
Setting up multi-broker cluster
Kafka in the purview of an SCV use case
When to use Kafka
When not to use Kafka
Other options
RabbitMQ
ZeroMQ
Apache ActiveMQ
Summary
Data Processing using Apache Flink
Context in a Data Lake - Data Ingestion Layer
Data Ingestion Layer
Data Ingestion Layer - technology mapping
What is Apache Flink?
Why Apache Flink?
History of Flink
Advantages of Flink
Disadvantages of Flink
Working of Flink
Flink architecture
Client
Job Manager
Task Manager
Flink execution model
Core architecture principles of Flink
Flink Component Stack
Checkpointing in Flink
Savepoints in Flink
Streaming window options in Flink
Time window
Count window
Tumbling window configuration
Sliding window configuration
Memory management
Flink API's
DataStream API
Flink DataStream API example
Streaming connectors
DataSet API
Flink DataSet API example
Table API
Flink domain specific libraries
Gelly - Flink Graph API
FlinkML
FlinkCEP
Flink working example
Installation
Example - data processing with Flink
Data generation
Step 1 - Preparing streams
Step 2 - Consuming Streams via Flink
Step 3 - Streaming data into HDFS
Flink in purview of SCV use cases
User Log Data Generation
Flume Setup
Flink Processors
When to use Flink
When not to use Flink
Other options
Apache Spark
Apache Storm
Apache Tez
Summary
Data Store Using Apache Hadoop
Context for Data Lake - Data Storage and lambda Batch layer
Data Storage and the Lambda Batch Layer
Data Storage and Lambda Batch Layer - technology mapping
What is Apache Hadoop?
Why Hadoop?
History of Hadoop
Advantages of Hadoop
Disadvantages of Hadoop
Working of Hadoop
Hadoop core architecture principles
Hadoop architecture
Hadoop architecture 1.x
Hadoop architecture 2.x
Hadoop architecture components
HDFS
YARN
MapReduce
Hadoop ecosystem
Hadoop architecture in detail
Hadoop ecosystem
Data access/processing components
Apache Pig
Apache Hive
Data storage components
Apache HBase
Monitoring, management and orchestration components
Apache ZooKeeper
Apache Oozie
Apache Ambari
Data integration components
Apache Sqoop
Apache Flume
Hadoop distributions
HDFS and formats
Hadoop for near real-time applications
Hadoop deployment modes
Hadoop working examples
Installation
Data preparation
Hive installation
Example - Bulk Data Load
File Data Load
RDBMS Data Load
Example - MapReduce processing
Text Data as Hive Tables
Avro Data as HIVE Table
Hadoop in purview of SCV use case
Initial directory setup
Data loads
Data visualization with HIVE tables
When not to use Hadoop
Other Hadoop Processing Options
Summary
Indexed Data Store using Elasticsearch
Context in Data Lake: data storage and lambda speed layer
Data Storage and Lambda Speed Layer
Data Storage and Lambda Speed Layer: technology mapping
What is Elasticsearch?
Why Elasticsearch
History of Elasticsearch
Advantages of Elasticsearch
Disadvantages of Elasticsearch
Working of Elasticsearch
Elasticsearch core architecture principles
Elasticsearch terminologies
Document in Elasticsearch
Index in Elasticsearch
What is Inverted Index?
Shard in Elasticsearch
Nodes in Elasticsearch
Cluster in Elasticsearch
Elastic Stack
Elastic Stack - Kibana
Elastic Stack - Elasticsearch
Elastic Stack - Logstash
Elastic Stack - Beats
Elastic Stack - X-Pack
Elastic Cloud
Apache Lucene
How Lucene works
Elasticsearch DSL (Query DSL)
Important queries in Query DSL
Nodes in Elasticsearch
Elasticsearch - master node
Elasticsearch - data node
Elasticsearch - client node
Elasticsearch and relational database
Elasticsearch ecosystem
Elasticsearch analyzers
Built-in analyzers
Custom analyzers
Elasticsearch plugins
Elasticsearch deployment options
Clients for Elasticsearch
Elasticsearch for fast streaming layer
Elasticsearch as a data source
Elasticsearch for content indexing
Elasticsearch and Hadoop
Elasticsearch working example
Installation
Creating and Deleting Indexes
Indexing Documents
Getting Indexed Document
Searching Documents
Updating Documents
Deleting a document
Elasticsearch in purview of SCV use case
Data preparation
Initial Cleanup
Data Generation
Customer data import into Hive using Sqoop
Data acquisition via Flume into Kafka channel
Data ingestion via Flink to HDFS and Elasticsearch
Packaging via POM file
Avro schema definitions
Schema deserialization class
Writing to HDFS as parquet files
Writing into Elasticsearch
Command line arguments
Flink deployment
Parquet data visualization as Hive tables
Data indexing from Hive
Query data from ES (customer, address, and contacts)
When to use Elasticsearch
When not to use Elasticsearch
Other options
Apache Solr
Summary
Data Lake Components Working Together
Where we stand with Data Lake
Core architecture principles of Data Lake
Challenges faced by enterprise Data Lake
Expectations from Data Lake
Data Lake for other activities
Knowing more about data storage
Zones in Data Storage
Data Schema and Model
Storage options
Apache HCatalog (Hive Metastore)
Compression methodologies
Data partitioning
Knowing more about Data processing
Data validation and cleansing
Machine learning
Scheduler/Workflow
Apache Oozie
Database setup and configuration
Build from Source
Oozie Workflows
Oozie coordinator
Complex event processing
Thoughts on data security
Apache Knox
Apache Ranger
Apache Sentry
Thoughts on data encryption
Hadoop key management server
Metadata management and governance
Metadata
Data governance
Data lineage
How can we achieve?
Apache Atlas
WhereHows
Thoughts on Data Auditing
Thoughts on data traceability
Knowing more about Serving Layer
Principles of Serving Layer
Service Types
GraphQL
Data Lake with REST API
Business services
Serving Layer components
Data Services
Elasticsearch & HBase
Apache Hive & Impala
RDBMS
Data exports
Polyglot data access
Example: serving layer
Summary
Data Lake Use Case Suggestions
Establishing cybersecurity practices in an enterprise
Know the customers dealing with your enterprise
Bring efficiency in warehouse management
Developing a brand and marketing of the enterprise
Achieve a higher degree of personalization with customers
Bringing IoT data analysis at your fingertips
More practical and useful data archival
Compliment the existing data warehouse infrastructure
Achieving telecom security and regulatory compliance
Summary
Preface
Data is becoming very important for many enterprises and it has now become pivotal in many aspects. In fact, companies are transforming themselves with data at the core. This book will start by introducing data, its relevance to enterprises, and how they can make use of this data to transform themselves digitally. To make use of data, enterprises need repositories, and in this modern age, these aren't called data warehouses; instead they are called Data Lake.
As we can see today, we have a good number of use cases that are leveraging big data technologies. The concept of a Data Lake existed there for quite sometime, but recently it has been getting real traction in enterprises. This book brings these two aspects together and gives a hand-on, full-fledged, working Data Lake using the latest big data technologies, following well-established architectural patterns.
The book will bring Data Lake and Lambda architecture together and help the reader to actually operationalize these in their enterprise. It will introduce a number of Big Data technologies at a high level, but we didn't want to make it an authoritative reference on any of these topics, as they are vast in nature and worthy of a book by themselves.
This book instead covers pattern explanation and implementation using chosen technologies. The technologies can of course, be replaced with more relevant ones in future or according to set standards within an organization. So, this book will be relevant not only now but for a long time to come. Compared to a software/technology written targeting a specific version, this does not fall in that category, so the shelf life of this book is quite long compared to other books in the same space.
The book will take you on a fantastic journey, and in doing so, it follows a structure that is quite intuitive and exciting at the same time.
What this book covers
The book is divided into three parts. Each part contains a few chapters, and when a part is completed, readers will have a clear picture of that part of the book in a holistic fashion. The parts are designed and structured in such a way that the reader is first introduced to major functional and technical aspects; then in the following part, or rather the final part, things will all come together. At the end of the book, readers will have an operational Data Lake.
Part 1, Overview, introduces the reader to various concepts relating to data, Data Lake and important components . It consists of four chapters and as detailed below, each chapter well-defined goal to be achieved.
Chapter 1, Introduction to Data, introduces the reader to the book in general and then explains what data is and its relevance to the enterprise. The chapter explains the reasons as to why data in modern world is important and how it can/should be used. Real-life use cases have been showcased to explain the significance of data and how data is transforming businesses today. These real-life use cases will help readers to start their creative juices flowing and get thinking about how they can make a difference to their enterprise using data.
Chapter 2, Comprehensive Concepts of a Data Lake, deepens further into the details of the concept of a Data Lake and explains use of Data Lake in addressing the problems faced by enterprises. This chapter also provides a sneak preview around Lambda architecture and how it can be leveraged for Data Lake. The reader would thus get introduced to the concept of a Data Lake and the various approaches that organizations have adopted to build Data Lake.
Chapter 3, Lambda Architecture as a Pattern for Data Lake, introduces the reader into details of Lambda architecture, its various components and the connection between Data Lake and this architecture pattern. In this chapter the reader will get details around Lambda architecture, with the reasons of its inception and the specific problems that it solves. The chapter also provides the reader with ability to understand the core concepts of Lambda architecture and how to apply it in an enterprise. The reader will understand various patterns and components that can be leveraged to define lambda architecture both in the batch and real-time processing spaces. The reader would have enough background on data, Data Lake and Lambda architecture by now, and can move onto the next section of implementing Data Lake for your enterprise.
Chapter 4, Applied Lambda for Data Lake, introduces reader to technologies which can be used for each layer (component) in Lambda architecture and will also help the reader choose one lead technology in the market which we feel very good at this point in time. In this chapter, the reader will understand various Hadoop distributions in the current landscape of Big Data technologies, and how they can be leveraged for applying Lambda architecture in an enterprise Data Lake. In the context of these technologies, the reader will understand the details of and architectural motivations behind batch, speed and serving layer in an enterprise Data Lake.
Part 2, Technical Building Blocks of Data Lake, introduces reader to many technologies which will be part of the Data Lake implementation. Each chapter covers a technology which will slowly build the Data Lake and the use case namely Single Customer View (SCV). Almost all the important technical details of the technology being discussed in each chapter would be covered in a holistic fashion as in-depth coverage is out of scope of this book. It consists of six chapters and each chapter has a goal well defined to be achieved as detailed below.
Chapter 5, Data Acquisition of Batch Data using Apache Sqoop, delves deep into Apache Sqoop. It gives reasons for this choice and also gives the reader other technology options with good amount of details. The chapter also gives a detailed example connecting Data Lake and Lambda architecture. In this chapter the reader will understand Sqoop framework and similar tools in the space for data loads from an enterprise data source into a Data Lake. The reader will understand the technical details around Sqoop and architecturally the problems that it solves. The reader will also be taken through examples, where the Sqoop will be seen in action and various steps involved in using it with Hadoop technologies.
Chapter 6, Data Acquisition of Stream Data using Apache Flume, delves deep into Apache Flume, thus connecting technologies in purview of Data Lake and Lambda architecture. The reader will understand Flume as a framework and its various patterns by which it can be leveraged for Data Lake. The reader will also understand the Flume architecture and technical details around using it to acquire and consume data using this framework in detail, with specific capabilities around transaction control and data replay with working example. The reader will also understand how to use flume with streaming technologies for stream based processing.
Chapter 7, Messaging Layer using Apache Kafka, delves deep into Apache Kafka. This part of the book initially gives the reader the reason for choosing a particular technology and also gives details of other technology options. . In this chapter, the reader would understand Kafka as a message oriented middleware and how it’s compared with other messaging engines. The reader will get to know details around Kafka and its functioning and how it can be leveraged for building scale-out capabilities, from the perspective of client (publisher), broker and consumer(subscriber). This reader will also understand how to integrate Kafka with Hadoop components for acquiring enterprise data and what capabilities this integration brings to Data Lake.
Chapter 8, Data Processing using Apache Flink, the reader in this chapter would understand the concepts around streaming and stream based processing, and specifically in reference to Apache Flink. The reader will get deep into using Apache Flink in context of Data Lake and in the Big Data technology landscape for near real time processing of data with working examples. The reader will also realize how a streaming functionality would depend on various other layers in architecture and how these layers can influence the streaming capability.
Chapter 9, Data Storage using Apache Hadoop, delves deep into Apache Hadoop. In this chapter, the reader would get deeper into Hadoop Landscape with various Hadoop components and their functioning and specific capabilities that these components can provide for an enterprise Data Lake. Hadoop in context of Data Lake is explained at an implementation level and how Hadoop frameworks capabilities around file storage, file formats and map-reduce capabilities can constitute the foundation for a Data Lake and specific patterns that can be applied to this stack for near real time capabilities.
Chapter 10, Indexed Data Store using Elasticsearch, delves deep into Elasticsearch. The reader will understand Elasticsearch as data indexing framework and various data analyzers provided by the framework for efficient searches. The reader will also understand how elasticsearch can be leveraged for Data Lake and data at scale with efficient sharding and distribution mechanisms for consistent performance. The reader will also understand how elasticsearch can be used for fast streaming and how it can used for high performance applications with working examples.
Part 3, Bringing it all together, will bring together technical components from part one and two of this book to give you a holistic picture of Data Lakes. We will bring in additional concepts and technologies in a brief fashion so that, if needed, you can explore those aspects in more detail according to your enterprise requirements. Again, delving deep into the technologies covered in this chapter is out of the scope of this book. But we want you to be aware of these additional technologies and how they can be brought into our Data Lake implementation if the need arises. It consists of two chapters, and each chapter has a goal well defined to be achieved, as detailed here.
Chapter 11, Data Lake components working together, after introducing reader into Data Lake, Lambda architecture, various technologies, this chapter brings the whole puzzle together and brings in a holistic picture to the reader. The reader at this stage should feel accomplished and can take in the codebase as is into the organization and show it working. In this chapter, the reader, would realize how to integrate various aspects of Data Lake to implement a fully functional Data Lake. The reader will also realize the completeness of Data Lake with working examples that would combine all the learning from previous chapters into a running implementation.
Chapter 12, Data Lake Use Case Suggestions, throughout the book the reader is taken through a use case in the form of Single Customer View
; however while going through the book, there are other use cases in pipeline relevant to their organization which reader can start thinking. This provoking of thought deepens into bit more during this chapter. The reader will understand and realize various use cases that can reap great benefits from a Data Lake and help optimize their cost of ownership, operations, reactiveness and help these uses with required intelligence derived from data. The reader, in this chapter, will also realize the variety of these use cases and the extents to which an enterprise Data Lake can be helpful for each of these use cases.
What you need for this book
This book is for developers, architects, and product/project owners, for realizing Lambda-architecture-based Data Lakes for Enterprises. This book comprises working examples to help the reader understand and observe various concepts around Data Lake and its basic implementation. In order to run the examples, one will need access to various pieces of open source software, required infrastructure, and development IDE. Specific efforts have been made to keep the examples simple and leverage commonly available frameworks and components. The operating system used for running these examples is CentOS 7, but these examples can run on any flavour of the Linux operating system.
Who this book is for
Java developers and architects who would like to implement Data Lake for their enterprise
Java developers who aim to get hands-on experience on Lambda Architecture and Big Data technologies
Java developers who would like to discover the world of Big Data and have an urge to implement a practical solution using those technologies.
Conventions
In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: Rename the completed spool file to spool-1 as specified in the earlier example.
A block of code is set as follows:
Any command-line input or output is written as follows:
New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: " Without minimum or no delay (NRT: Near Real Time or Real time) the company wanted the data produced to be moved to Hadoop system"
Warnings or important notes appear in a box like this.
Tips and tricks appear like this.
Reader feedback
Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.
To send us general feedback, simply e-mail [email protected], and mention the book's title in the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Customer support
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.