Data Lake Development with Big Data
()
About this ebook
Related to Data Lake Development with Big Data
Related ebooks
Data Engineering on Azure Rating: 0 out of 5 stars0 ratingsDesigning Cloud Data Platforms Rating: 0 out of 5 stars0 ratingsTesting the Data Warehouse Practicum: Assuring Data Content, Data Structures and Quality Rating: 0 out of 5 stars0 ratingsPrinciples of Data Management: Facilitating information sharing Rating: 0 out of 5 stars0 ratingsMLOps Engineering at Scale Rating: 0 out of 5 stars0 ratingsSmarter Data Science: Succeeding with Enterprise-Grade Data and AI Projects Rating: 0 out of 5 stars0 ratingsPostgreSQL for Data Architects Rating: 0 out of 5 stars0 ratingsOracle Warehouse Builder 11g: Getting Started Rating: 0 out of 5 stars0 ratingsData Quality: Empowering Businesses with Analytics and AI Rating: 0 out of 5 stars0 ratingsData Analytics with Python: Data Analytics in Python Using Pandas Rating: 3 out of 5 stars3/5Mastering Python for Data Science Rating: 3 out of 5 stars3/5Data Lake for Enterprises Rating: 0 out of 5 stars0 ratingsArchitecting Big Data & Analytics Solutions - Integrated with IoT & Cloud Rating: 5 out of 5 stars5/5Data Lake Architecture Complete Self-Assessment Guide Rating: 0 out of 5 stars0 ratingsReal-Time Big Data Analytics Rating: 5 out of 5 stars5/5Learning PySpark Rating: 0 out of 5 stars0 ratingsFrank Kane's Taming Big Data with Apache Spark and Python Rating: 0 out of 5 stars0 ratingsLearn Data Warehousing in 24 Hours Rating: 0 out of 5 stars0 ratingsData Pipelines with Apache Airflow Rating: 0 out of 5 stars0 ratingsBig Data Analytics Rating: 0 out of 5 stars0 ratingsHadoop Real-World Solutions Cookbook - Second Edition Rating: 0 out of 5 stars0 ratingsDatabricks A Complete Guide - 2021 Edition Rating: 0 out of 5 stars0 ratingsThe Data Model Resource Book: Volume 3: Universal Patterns for Data Modeling Rating: 0 out of 5 stars0 ratingsData warehouse Complete Self-Assessment Guide Rating: 4 out of 5 stars4/5Mastering Data Warehouse Design: Relational and Dimensional Techniques Rating: 4 out of 5 stars4/5Google Cloud Platform for Data Engineering: From Beginner to Data Engineer using Google Cloud Platform Rating: 5 out of 5 stars5/5
Enterprise Applications For You
QuickBooks 2023 All-in-One For Dummies Rating: 0 out of 5 stars0 ratingsExcel Formulas That Automate Tasks You No Longer Have Time For Rating: 5 out of 5 stars5/5Creating Online Courses with ChatGPT | A Step-by-Step Guide with Prompt Templates Rating: 4 out of 5 stars4/5Excel : The Ultimate Comprehensive Step-By-Step Guide to the Basics of Excel Programming: 1 Rating: 5 out of 5 stars5/5Bitcoin For Dummies Rating: 4 out of 5 stars4/5QuickBooks 2024 All-in-One For Dummies Rating: 0 out of 5 stars0 ratingsEnterprise AI For Dummies Rating: 3 out of 5 stars3/550 Useful Excel Functions: Excel Essentials, #3 Rating: 5 out of 5 stars5/5QuickBooks Online For Dummies Rating: 0 out of 5 stars0 ratingsExcel 2016 For Dummies Rating: 4 out of 5 stars4/5Access 2019 For Dummies Rating: 0 out of 5 stars0 ratingsExcel Formulas and Functions 2020: Excel Academy, #1 Rating: 4 out of 5 stars4/5Microsoft 365 For Dummies Rating: 0 out of 5 stars0 ratingsScrivener For Dummies Rating: 4 out of 5 stars4/5Excel 2019 For Dummies Rating: 3 out of 5 stars3/5ChatGPT Ultimate User Guide - How to Make Money Online Faster and More Precise Using AI Technology Rating: 1 out of 5 stars1/5Mastering QuickBooks 2020: The ultimate guide to bookkeeping and QuickBooks Online Rating: 0 out of 5 stars0 ratingsExcel Data Analysis For Dummies Rating: 0 out of 5 stars0 ratingsNotion for Beginners: Notion for Work, Play, and Productivity Rating: 4 out of 5 stars4/5Improve your skills with Google Sheets: Professional training Rating: 0 out of 5 stars0 ratings101 Ready-to-Use Excel Formulas Rating: 4 out of 5 stars4/5Create Income through Self-Publishing: An Author's Approach on Generating Wealth by Self-Publishing Rating: 5 out of 5 stars5/5Managing Humans: Biting and Humorous Tales of a Software Engineering Manager Rating: 4 out of 5 stars4/5
Reviews for Data Lake Development with Big Data
0 ratings0 reviews
Book preview
Data Lake Development with Big Data - Pasupuleti Pradeep
Table of Contents
Data Lake Development with Big Data
Credits
About the Authors
Acknowledgement
About the Reviewer
www.PacktPub.com
Support files, eBooks, discount offers, and more
Why subscribe?
Free access for Packt account holders
Preface
What this book covers
What you need for this book
Who this book is for
Conventions
Reader feedback
Customer support
Errata
Piracy
Questions
1. The Need for Data Lake
Before the Data Lake
Need for Data Lake
Defining Data Lake
Key benefits of Data Lake
Challenges in implementing a Data Lake
When to go for a Data Lake implementation
Data Lake architecture
Architectural considerations
Architectural composition
Architectural details
Understanding Data Lake layers
The Data Governance and Security Layer
The Information Lifecycle Management layer
The Metadata Layer
Understanding Data Lake tiers
The Data Intake tier
The Source System Zone
The Transient Zone
The Raw Zone
Batch Raw Storage
The real-time Raw Storage
The Data Management tier
The Integration Zone
The Enrichment Zone
The Data Hub Zone
The Data Consumption tier
The Data Discovery Zone
The Data Provisioning Zone
Summary
2. Data Intake
Understanding Intake tier zones
Source System Zone functionalities
Understanding connectivity processing
Understanding Intake Processing for data variety
Structured data
The need for integrating Structured Data in the Data Lake
Structured data loading approaches
Semi-structured data
The need for integrating semi-structured data in the Data Lake
Semi-structured data loading approaches
Unstructured data
The need for integrating Unstructured data in the Data Lake
Unstructured data loading approaches
Transient Landing Zone functionalities
File validation checks
File duplication checks
File integrity checks
File size checks
File periodicity checks
Data Integrity checks
Checking record counts
Checking for column counts
Schema validation checks
Raw Storage Zone functionalities
Data lineage processes
Watermarking process
Metadata capture
Deep Integrity checks
Bit Level Integrity checks
Periodic checksum checks
Security and governance
Information Lifecycle Management
Practical Data Ingestion scenarios
Architectural guidance
Structured data use cases
Semi-structured and unstructured data use cases
Big Data tools and technologies
Ingestion of structured data
Sqoop
Use case scenarios for Sqoop
WebHDFS
Use case scenarios for WebHDFS
Ingestion of streaming data
Apache Flume
Use case scenarios for Flume
Fluentd
Use case scenarios for Fluentd
Kafka
Use case scenarios for Kafka
Amazon Kinesis
Use case scenarios for Kinesis
Apache Storm
Use case scenarios for Storm
Summary
3. Data Integration, Quality, and Enrichment
Introduction to the Data Management Tier
Understanding Data Integration
Introduction to Data Integration
Prominent features of Data Integration
Loosely coupled Integration
Ease of use
Secure access
High-quality data
Lineage tracking
Practical Data Integration scenarios
The workings of Data Integration
Raw data discovery
Data quality assessment
Profiling the data
Data cleansing
Deletion of missing, null, or invalid values
Imputation of missing, null, or invalid values
Data transformations
Unstructured text transformation techniques
Structured data transformations
Data enrichment
Collect metadata and track data lineage
Traditional Data Integration versus Data Lake
Data pipelines
Addressing the limitations using Data Lake
Data partitioning
Addressing the limitations using Data Lake
Scale on demand
Addressing the limitations using Data Lake
Data ingest parallelism
Addressing the limitations using Data Lake
Extensibility
Addressing the limitations using Data Lake
Big Data tools and technologies
Syncsort
Use case scenarios for Syncsort
Talend
Use case scenarios for Talend
Pentaho
Use case scenarios for Pentaho
Summary
4. Data Discovery and Consumption
Understanding the Data Consumption tier
Data Consumption – Traditional versus Data Lake
An introduction to Data Consumption
Practical Data Consumption scenarios
Data Discovery and metadata
Enabling Data Discovery
Data classification
Classifying unstructured data
Named entity recognition
Topic modeling
Text clustering
Applications of data classification
Relation extraction
Extracting relationships from unstructured data
Feature-based methods
Understanding how feature-based methods work
Implementation
Semantic technologies
Understanding how semantic technologies work
Implementation
Extracting Relationships from structured data
Applications of relation extraction
Indexing data
Inverted index
Understanding how inverted index works
Implementation
Applications of Indexing
Performing Data Discovery
Semantic search
Word sense disambiguation
Latent Semantic Analysis
Faceted search
Fuzzy search
Edit distance
Wildcard and regular expressions
Data Provisioning and metadata
Data publication
Data subscription
Data Provisioning functionalities
Data formatting
Data selection
Data Provisioning approaches
Post-provisioning processes
Architectural guidance
Data Discovery
Big Data tools and technologies
Elasticsearch
Use case scenarios for Elasticsearch
IBM InfoSphere Data Explorer
Use case scenarios for IBM InfoSphere Data Explorer
Tableau
Use case scenarios for Tableau
Splunk
Use case scenarios for Splunk
Data Provisioning
Big Data tools and technologies
Data Dispatch
Use case scenarios for Data Dispatch
Summary
5. Data Governance
Understanding Data Governance
Introduction to Data Governance
The need for Data Governance
Governing Big Data in the Data Lake
Data Governance – Traditional versus Data Lake
Practical Data Governance scenarios
Data Governance components
Metadata management and lineage tracking
Data security and privacy
Big Data implications for security and privacy
Security issues in the Data Lake tiers
The Intake Tier
The Management Tier
The Consumption Tier
Information Lifecycle Management
Big Data implications for ILM
Implementing ILM using Data Lake
The Intake Tier
The Management Tier
The Consumption Tier
Architectural guidance
Big Data tools and technologies
Apache Falcon
Understanding how Falcon works
Use case scenarios for Falcon
Apache Atlas
Understanding how Atlas works
Use case scenarios for Atlas
IBM Big Data platform
Understanding how governance is provided in IBM Big Data platform
Use case scenarios for IBM Big Data platform
The current and future trends
Data Lake and future enterprise trajectories
Future Data Lake technologies
Summary
Index
Data Lake Development with Big Data
Data Lake Development with Big Data
Copyright © 2015 Packt Publishing
All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.
Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.
Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.
First published: November 2015
Production reference: 1241115
Published by Packt Publishing Ltd.
Livery Place
35 Livery Street
Birmingham B3 2PB, UK.
ISBN 978-1-78588-808-3
www.packtpub.com
Credits
Authors
Pradeep Pasupuleti
Beulah Salome Purra
Reviewer
Dr. Kornel Amadeusz Skałkowski
Commissioning Editor
Priya Singh
Acquisition Editor
Ruchita Bhansali
Content Development Editor
Rohit Kumar Singh
Technical Editor
Saurabh Malhotra
Copy Editor
Trishya Hajare
Project Coordinator
Izzat Contractor
Proofreader
Safis Editing
Indexer
Hemangini Bari
Graphics
Jason Monteiro
Kirk D'Penha
Production Coordinator
Shantanu N. Zagade
Cover Work
Shantanu N. Zagade
About the Authors
Pradeep Pasupuleti has 18 years of experience in architecting and developing distributed and real-time data-driven systems. He constantly explores ways to use the power and promise of advanced analytics-driven platforms to solve the problems of the common man. He founded Datatma, a consulting firm, with a mission to humanize Big Data analytics, putting it to use to solve simple problems that serve a higher purpose.
He architected robust Big Data-enabled automated learning engines that enterprises regularly use in production in order to save time, money, and the lives of humans.
He built solid interdisciplinary data science teams that bridged the gap between theory and practice, thus, creating compelling data products. His primary focus is always to ensure his customers are delighted by assisting and addressing their business problems through data products that use Big Data technologies and algorithms. He consistently demonstrated thought leadership by solving high-dimensional data problems and getting phenomenal results.
He has performed strategic leadership roles in technology consulting, advising Fortune 100 companies on Big Data strategy and creating Big Data Centers of Excellence.
He has worked on use cases such as enterprise Data Lake, fraud detection, patient re-admission prediction, student performance prediction, claims optimization sentiment mining, cloud infrastructure SLA violation prediction, data leakage prevention, and mainframe offloaded ETL on Hadoop.
In the book Pig Design Patterns, Packt Publishing, he has compiled his learning and experiences from the challenges involved in building Hadoop-driven data products such as data ingest, data cleaning and validating, data transformation, dimensionality reduction, and many other interesting Big Data war stories.
Out of his office hours, he enjoys running marathons, exploring archeological sites, finding patterns in unrelated data sources, mentoring start-ups, and budding researchers.
He can be reached at <[email protected]> and https://in.linkedin.com/in/pradeeppasupuleti.
Acknowledgement
This book is dedicated to the loving memory of my mother, Smt. Sumathy; without her never-failing encouragement and everlasting love I would have never been half as good.
First and foremost, I have to thank my father, Sri. Prabhakar Pasupuleti, who never ceases to be a constant source of inspiration, a ray of hope, humility and strength, and whose support and guidance have given me the courage to chase my dreams.
I should also express my deep sense of gratitude to each of my family members, Sushma, Sresht, and Samvruth, who stood by me at every moment through very tough times and enabled me to complete this book.
I would like to sincerely thank all my teachers who were instrumental in shaping me. Among them, I would like to thank Usha Madam, Vittal Rao Sir, Gopal Krishna Sir, and Brindavan Sir for their stellar role in improving me.
I would also like to thank all my friends for their understanding in many ways. Their friendship makes my life a wonderful experience. I cannot list all the names here, but you are always on my mind.
Special thanks to the team at Packt for their contribution to this book.
Finally, I would like to thank my team, Salome et.al, that has placed immense faith in the power of Big Data analytics and built cutting edge data products.
Thank you lord, for always being there for me.
Beulah Salome Purra has over 11 years of experience and she specializes in building highly scalable distributed systems. She has worked extensively on architecting multiple large-scale Big Data solutions for Fortune 100 companies. Her core expertise lies in working on Big Data Analytics. In her current role at ATMECS, her focus is on building robust and scalable data products that extract value from huge data assets.
She can be reached at https://www.linkedin.com/in/beulahsalomep.
I am grateful to my parents, Rathnam and Padma, who have constantly encouraged and supported me throughout. I would like to thank my husband, Pratap, for his help on this book, his patience, love, and support; my brothers, Joel and Michael, for all their support.
I would like to profusely thank Pradeep Pasupuleti for mentoring me; working with him has been an enriching experience. I can't thank him enough for his constant encouragement, guidance, support, and for providing me an opportunity to work with him on this book.
Special thanks to David Hawke, Sanjay Singh, and Ravi Velagapudi—the leadership team at ATMECS—for their encouragement and support while I was writing this book.
Thanks to the editors and reviewers at Packt for all their effort in making this book better.
About the Reviewer
Dr. Kornel Amadeusz Skałkowski has a solid academic and industrial background. For more than 5 years, he worked as an assistant at AGH University of Science and Technology in Krakow. In 2015, he obtained his PhD. in the subject of machine learning-based adaptation of the SOA systems. He has cooperated with several companies on various projects concerning intelligent systems, machine learning, and Big Data. Currently, he works as a Big Data developer for SAP SE.
He is the co-author of 19 papers concerning software engineering, SOA systems, and machine learning. He also works as a reviewer for the American Journal of Software Engineering and Applications. He has participated in numerous European and national scientific projects. His research interests include machine learning, Big Data, and software engineering.
I would like to kindly thank my family, relatives, and friends, for their endless patience and support during the reviewing of this book.
www.PacktPub.com
Support files, eBooks, discount offers, and more
For support files and downloads related to your book, please visit www.PacktPub.com.
Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at
At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.
https://www2.packtpub.com/books/subscription/packtlib
Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.
Why subscribe?
Fully searchable across every book published by Packt
Copy and paste, print, and bookmark content
On demand and accessible via a web browser
Free access for Packt account holders
If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view 9 entirely free books. Simply use your login credentials for immediate access.
Preface
The book Data Lake Development with Big Data is a practical guide to help you learn the essential architectural approaches to design and build Data Lakes. It walks you through the various components of Data Lakes, such as data