Data Engineering on Azure
()
About this ebook
Summary
In Data Engineering on Azure you will learn how to:
Pick the right Azure services for different data scenarios
Manage data inventory
Implement production quality data modeling, analytics, and machine learning workloads
Handle data governance
Using DevOps to increase reliability
Ingesting, storing, and distributing data
Apply best practices for compliance and access control
Data Engineering on Azure reveals the data management patterns and techniques that support Microsoft’s own massive data infrastructure. Author Vlad Riscutia, a data engineer at Microsoft, teaches you to bring an engineering rigor to your data platform and ensure that your data prototypes function just as well under the pressures of production. You'll implement common data modeling patterns, stand up cloud-native data platforms on Azure, and get to grips with DevOps for both analytics and machine learning.
Purchase of the print book includes a free eBook in PDF, Kindle, and ePub formats from Manning Publications.
About the technology
Build secure, stable data platforms that can scale to loads of any size. When a project moves from the lab into production, you need confidence that it can stand up to real-world challenges. This book teaches you to design and implement cloud-based data infrastructure that you can easily monitor, scale, and modify.
About the book
In Data Engineering on Azure you’ll learn the skills you need to build and maintain big data platforms in massive enterprises. This invaluable guide includes clear, practical guidance for setting up infrastructure, orchestration, workloads, and governance. As you go, you’ll set up efficient machine learning pipelines, and then master time-saving automation and DevOps solutions. The Azure-based examples are easy to reproduce on other cloud platforms.
What's inside
Data inventory and data governance
Assure data quality, compliance, and distribution
Build automated pipelines to increase reliability
Ingest, store, and distribute data
Production-quality data modeling, analytics, and machine learning
About the reader
For data engineers familiar with cloud computing and DevOps.
About the author
Vlad Riscutia is a software architect at Microsoft.
Table of Contents
1 Introduction
PART 1 INFRASTRUCTURE
2 Storage
3 DevOps
4 Orchestration
PART 2 WORKLOADS
5 Processing
6 Analytics
7 Machine learning
PART 3 GOVERNANCE
8 Metadata
9 Data quality
10 Compliance
11 Distributing data
Vlad Riscutia
Vlad Riscutia is a software architect at Microsoft.
Related to Data Engineering on Azure
Related ebooks
Azure Storage, Streaming, and Batch Analytics: A guide for data engineers Rating: 0 out of 5 stars0 ratingsData Pipelines with Apache Airflow Rating: 0 out of 5 stars0 ratingsDesigning Cloud Data Platforms Rating: 0 out of 5 stars0 ratingsData Lake Development with Big Data Rating: 0 out of 5 stars0 ratingsMLOps Engineering at Scale Rating: 0 out of 5 stars0 ratingsGoogle Cloud Platform for Data Engineering: From Beginner to Data Engineer using Google Cloud Platform Rating: 5 out of 5 stars5/5Hands-on Cloud Analytics with Microsoft Azure Stack Rating: 0 out of 5 stars0 ratingsLearning Microsoft Azure Rating: 4 out of 5 stars4/5Effective Data Science Infrastructure: How to make data scientists productive Rating: 0 out of 5 stars0 ratingsThe Definitive Guide to Azure Data Engineering: Modern ELT, DevOps, and Analytics on the Azure Cloud Platform Rating: 0 out of 5 stars0 ratingsExpert Cube Development with SSAS Multidimensional Models Rating: 0 out of 5 stars0 ratingsBuilding Web Services with Microsoft Azure Rating: 0 out of 5 stars0 ratingsInfrastructure as Code, Patterns and Practices: With examples in Python and Terraform Rating: 0 out of 5 stars0 ratingsSQL Server Data Automation Through Frameworks: Building Metadata-Driven Frameworks with T-SQL, SSIS, and Azure Data Factory Rating: 0 out of 5 stars0 ratingsAI as a Service: Serverless machine learning with AWS Rating: 1 out of 5 stars1/5Data Lake for Enterprises Rating: 0 out of 5 stars0 ratingsData Lake Architecture Complete Self-Assessment Guide Rating: 0 out of 5 stars0 ratingsAzure Databricks A Complete Guide - 2019 Edition Rating: 0 out of 5 stars0 ratingsData Analysis with Python and PySpark Rating: 0 out of 5 stars0 ratingsFrank Kane's Taming Big Data with Apache Spark and Python Rating: 0 out of 5 stars0 ratingsOfficial Google Cloud Certified Professional Data Engineer Study Guide Rating: 5 out of 5 stars5/5Databricks A Complete Guide - 2021 Edition Rating: 0 out of 5 stars0 ratingsHow to Lead in Data Science Rating: 0 out of 5 stars0 ratingsLearning PySpark Rating: 0 out of 5 stars0 ratingsData Analytics with Google Cloud Platform: Build Real Time Data Analytics on Google Cloud Platform Rating: 0 out of 5 stars0 ratings
Programming For You
Excel : The Ultimate Comprehensive Step-By-Step Guide to the Basics of Excel Programming: 1 Rating: 5 out of 5 stars5/5SQL QuickStart Guide: The Simplified Beginner's Guide to Managing, Analyzing, and Manipulating Data With SQL Rating: 4 out of 5 stars4/5Learn to Code. Get a Job. The Ultimate Guide to Learning and Getting Hired as a Developer. Rating: 5 out of 5 stars5/5Coding All-in-One For Dummies Rating: 4 out of 5 stars4/5Python Programming : How to Code Python Fast In Just 24 Hours With 7 Simple Steps Rating: 4 out of 5 stars4/5Grokking Algorithms: An illustrated guide for programmers and other curious people Rating: 4 out of 5 stars4/5HTML & CSS: Learn the Fundaments in 7 Days Rating: 4 out of 5 stars4/5Assembly Programming:Simple, Short, And Straightforward Way Of Learning Assembly Language Rating: 5 out of 5 stars5/5Learn JavaScript in 24 Hours Rating: 3 out of 5 stars3/5HTML in 30 Pages Rating: 5 out of 5 stars5/5SQL All-in-One For Dummies Rating: 3 out of 5 stars3/5PYTHON: Practical Python Programming For Beginners & Experts With Hands-on Project Rating: 5 out of 5 stars5/5Spies, Lies, and Algorithms: The History and Future of American Intelligence Rating: 4 out of 5 stars4/5Python Data Structures and Algorithms Rating: 5 out of 5 stars5/5C# Programming from Zero to Proficiency (Beginner): C# from Zero to Proficiency, #2 Rating: 0 out of 5 stars0 ratingsC All-in-One Desk Reference For Dummies Rating: 5 out of 5 stars5/5Hacking: Ultimate Beginner's Guide for Computer Hacking in 2018 and Beyond: Hacking in 2018, #1 Rating: 4 out of 5 stars4/5Python: For Beginners A Crash Course Guide To Learn Python in 1 Week Rating: 4 out of 5 stars4/5Raspberry Pi Electronics Projects for the Evil Genius Rating: 3 out of 5 stars3/5Windows 11 For Dummies Rating: 0 out of 5 stars0 ratings
Reviews for Data Engineering on Azure
0 ratings0 reviews
Book preview
Data Engineering on Azure - Vlad Riscutia
inside front cover
Data Platform Architecture
Architecture of a big data platform with the Azure services used in the reference implementation presented in this book
Data is ingested into the system and persisted in a storage layer. Processing aggregates and reshapes the data to enable analytics and machine learning scenarios. Orchestration and governance are cross-cutting concerns that cover all the components of the platform. Once processed, data is distributed to other downstream systems. All components are tracked by and deployed from source control.
Data Engineering on Azure
Vlad Riscutia
To comment go to liveBook
Manning
Shelter Island
For more information on this and other Manning titles go to
www.manning.com
Copyright
For online information and ordering of these and other Manning books, please visit www.manning.com. The publisher offers discounts on these books when ordered in quantity.
For more information, please contact
Special Sales Department
Manning Publications Co.
20 Baldwin Road
PO Box 761
Shelter Island, NY 11964
Email: [email protected]
©2021 by Manning Publications Co. All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.
Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps.
♾ Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine.
ISBN: 9781617298929
dedication
To my daughter, Ada
brief contents
1 Introduction
Part 1 Infrastructure
2 Storage
3 DevOps
4 Orchestration
Part 2 Workloads
5 Processing
6 Analytics
7 Machine learning
Part 3 Governance
8 Metadata
9 Data quality
10 Compliance
11 Distributing data
Appendix A. Azure services
Appendix B. KQL quick reference
Appendix C. Running code samples
contents
preface
acknowledgments
about this book
about the author
about the cover illustration
1 Introduction
1.1 What is data engineering?
1.2 Who this book is for
1.3 What is a data platform?
Anatomy of a data platform
Infrastructure as code, codeless infrastructure
1.4 Building in the cloud
IaaS, PaaS, SaaS
Network, storage, compute
Getting started with Azure
Interacting with Azure
1.5 Implementing an Azure data platform
Part 1 Infrastructure
2 Storage
2.1 Storing data in a data platform
Storing data across multiple data fabrics
Having a single source of truth
2.2 Introducing Azure Data Explorer
Deploying an Azure Data Explorer cluster
Using Azure Data Explorer
Working around query limits
2.3 Introducing Azure Data Lake Storage
Creating an Azure Data Lake Storage account
Using Azure Data Lake Storage
Integrating with Azure Data Explorer
2.4 Ingesting data
Ingestion frequency
Load type
Restatements and reloads
3 DevOps
3.1 What is DevOps?
DevOps in data engineering
3.2 Introducing Azure DevOps
Using the az azure-devops extension
3.3 Deploying infrastructure
Exporting an Azure Resource Manager template
Creating Azure DevOps service connections
Deploying Azure Resource Manager templates
Understanding Azure Pipelines
3.4 Deploying analytics
Using Azure DevOps marketplace extensions
Storing everything in Git; deploying everything automatically
4 Orchestration
4.1 Ingesting the Bing COVID-19 open dataset
4.2 Introducing Azure Data Factory
Setting up the data source
Setting up the data sink
Setting up the pipeline
Setting up a trigger
Orchestrating with Azure Data Factory
4.3 DevOps for Azure Data Factory
Deploying Azure Data Factory from Git
Setting up access control
Deploying the production data factory
DevOps for the Azure Data Factory recap
4.4 Monitoring with Azure Monitor
Part 2 Workloads
5 Processing
5.1 Data modeling techniques
Normalization and denormalization
Data warehousing
Semistructured data
Data modeling recap
5.2 Identity keyrings
Building an identity keyring
Understanding keyrings
5.3 Timelines
Building a timeline view
Using timelines
5.4 Continuous data processing
Tracking processing functions in Git
Keyring building in Azure Data Factory
Scaling out
6 Analytics
6.1 Structuring storage
Providing development data
Replicating production data
Providing read-only access to the production data
Storage structure recap
6.2 Analytics workflow
Prototyping
Development and user acceptance testing
Production
Analytics workflow recap
6.3 Self-serve data movement
Support model
Data contracts
Pipeline validation
Postmortems
Self-serve data movement recap
7 Machine learning
7.1 Training a machine learning model
Training a model using scikit-learn
High spender model implementation
7.2 Introducing Azure Machine Learning
Creating a workspace
Creating an Azure Machine Learning compute target
Setting up Azure Machine Learning storage
Running ML in the cloud
Azure Machine Learning recap
7.3 MLOps
Deploying from Git
Storing pipeline IDs
DevOps for Azure Machine Learning recap
7.4 Orchestrating machine learning
Connecting Azure Data Factory with Azure Machine Learning
Machine learning orchestration
Orchestrating recap
Part 3 Governance
8 Metadata
8.1 Making sense of the data
8.2 Introducing Azure Purview
8.3 Maintaining a data inventory
Setting up a scan
Browsing the data dictionary
Data dictionary recap
8.4 Managing a data glossary
Adding a new glossary term
Curating terms
Custom templates and bulk import
Data glossary recap
8.5 Understanding Azure Purview's advanced features
Tracking lineage
Classification rules
REST API
Advanced features recap
9 Data quality
9.1 Testing data
Availability tests
Correctness tests
Completeness tests
Detecting anomalies
Testing data recap
9.2 Running data quality checks
Testing using Azure Data Factory
Executing tests
Creating and using a template
Running data quality checks recap
9.3 Scaling out data testing
Supporting multiple data fabrics
Testing at rest and during movement
Authoring tests
Storing tests and results
10 Compliance
10.1 Data classification
Feature data
Telemetry
User data
User-owned data
Business data
Data classification recap
10.2 Changing classification through processing
Aggregation
Anonymization
Pseudonymization
Masking
Processing classification changes recap
10.3 Implementing an access model
Security groups
Securing Azure Data Explorer
Access model recap
10.4 Complying with GDPR and other considerations
Data handling
Data subject requests
Other considerations
11 Distributing data
11.1 Data distribution overview
11.2 Building a data API
Introducing Azure Cosmos DB
Populating the Cosmos DB collection
Retrieving data
Data API recap
11.3 Serving machine learning
11.4 Sharing data for bulk copy
Separating compute resources
Introducing Azure Data Share
Sharing data for bulk copy recap
11.5 Data sharing best practices
Appendix A. Azure services
Appendix B. KQL quick reference
Appendix C. Running code samples
index
front matter
preface
This is the book I wish I had available to refer to over the past few years, while scaling out the big data platform of the Customer Growth and Analytics team in Azure. As our data science team grew and the insights generated by the team became more and more critical to the business, we had to ensure that our platform was robust.
The world of big data is relatively new, and the playbook is still being written. I believe our story is common: data teams start small with a handful of people, who first prove they can generate valuable insights. At this stage, a lot of work happens ad hoc, and there is no immediate need for big engineering investments. A data scientist can run a machine learning (ML) model on their machine, generate some predictions, and email the results.
Over time, the team grows and more workloads become mission critical. The same ML model now plugs into a system serving live traffic and needs to run on a daily basis with more than a hundred times the data it was originally prototyped with. At this point, solid engineering practices are critical; we need scale, reliability, automation, monitoring, etc.
This book contains several years of hard-learned lessons in data engineering. To name a few examples:
Empowering every data scientist on the team to deploy new analytics and data movement pipelines onto our platform while maintaining a reliable production environment
Architecting an ML platform to streamline and automate execution of dozens of ML models
Building a metadata catalog to make sense of the large number of available datasets
Implementing various ways to test the quality of the data and sending alerts when issues are identified
The underlying theme of this book is DevOps, bringing the decades-old best practices of software engineering to the world of big data. Data governance is another important topic; making sense of the data, ensuring quality, compliance, and access control are all a critical part of governance.
The patterns and practices described in this book are platform agnostic. They should be just as valid regardless of which cloud you use. That said, we can’t be too abstract, so I provide some concrete examples through a reference implementation. The reference implementation is Azure. Even here, there is a wide selection of services we can pick from.
The reference implementation uses a set of services, but keep in mind, the book is less about the particular set of services and more about the data engineering practices realized through them. I hope you enjoy the book, and that you find some best practices you can apply to your environment and business space.
acknowledgments
Many thanks to my wife, Diana, and daughter, Ada, for their support. Thanks for bearing with me for a second round!
This book wouldn’t be what it is without the great input and advice from Michael Stephens and Elesha Hyde. Also, thanks go to Danny Vinson for reviewing the early draft and to Karsten Strøbæk for checking all the code samples. I thank all the reviewers for their time and feedback: Albert Nogués, Arun Thangasamy, Dave Corun, Geoff Clark, Glenn Swonk, Hilde Van Gysel, Jesús A. Juárez Guerrero, Johannes Verwijnen, Kelum Senanayake, Krzysztof Kamyczek, Luke Kupka, Matthias Busch, Miranda Whurr, Oliver Korten, Peter Kreyenhop, Peter Morgan, Phil Allen, Philippe Van Bergen, Richard B. Ward, Richard Vaughan, Robert Walsh, Sven Stumpf, Todd Cook, Vishwesh Ravi Shrimali, and Zekai Otles.
Many thanks go to the Customer Growth and Analytics leadership team for their support and for giving me the opportunity to learn: Tim Wong, Greg Koehler, Ron Sielinski, Merav Davidson, Vivek Dalvi, and everyone else on the team.
I was also fortunate to partner with many other teams across Microsoft. I want to thank the IDEAs team, especially Gerardo Bodegas Martinez, Wayne Yim, and Ayyappan Balasubramanian; the Azure Data Explorer team, Oded Sacher and Ziv Caspi; the Azure Purview team, Naga Krishna Yenamandra and Gaurav Malhotra; and the Azure Machine Learning team, especially Tzvi Keisar.
And I thank the Manning team, who helped put this book together from development through production and everything in between.
about this book
Just as software engineering brings engineering rigor to software development, data engineering aims to bring the same rigor to working with data in a reliable way. This book is about implementing the various aspects of a big data platform in a real-world production system: data ingestion, running analytics and machine learning (ML), and distributing data, to name a few. The focus of this book is on the operational aspects such as DevOps, monitoring, scale, and compliance. Examples are provided using Azure services.
Who should read this book?
A typical reader is a data scientist, software engineer, or architect with several years of experience who has become a data engineer looking into building and scaling a production data platform. Readers should have a basic knowledge of the cloud and some experience working with data.
How this book is organized: A roadmap
This book is divided into three parts, and each part looks at a data platform through a different lens. Chapter 1 introduces the overall architecture of a data platform, gives an overview of the Azure services we’ll use for the reference implementation, and defines some of the key terms (such as what we mean by data engineering and infrastructure as code, etc.) to lay some common groundwork. Then, part 1 covers the core infrastructure of a data platform:
Chapter 2 discusses storage infrastructure, the heart of a big data platform.
Chapter 3 covers DevOps, the key ingredient that brings engineering discipline to the realm of data.
Chapter 4 talks about orchestration, how data movement and processing is scheduled and executed throughout the platform.
Part 2 covers the main workloads a data platform needs to support:
Chapter 5 deals with processing data, reshaping it to better support various analytical scenarios.
Chapter 6 covers analytics and how we can apply good engineering practices to recurring reporting and analysis.
Chapter 7 shows how we can support end-to-end machine learning workloads (also known as MLOps).
Part 3 cover various aspects of governance:
Chapter 8 focuses on metadata (data about the data) and how to make sense of all the assets in a big data platform.
Chapter 9 discusses data quality and different types of tests that we can run against our datasets.
Chapter 10 covers an important topic—compliance—including how we classify and handle different types of data.
Chapter 11 talks about data distribution and the various way data is shared with other teams downstream.
The chapters can be read in any order, as these each touch on different aspects of data engineering. Part 1, however, is a prerequisite if you want to run the code examples. These chapters also set up the foundational pieces of the infrastructure, but otherwise, feel free to skip around and focus on the chapters that sound most interesting to you.
About the code
This book contains many examples of source code both in numbered listings and in line with normal text. In both cases, source code is formatted in a fixed-width font, like this to separate it from ordinary text.
Also, in many cases, the original source code has been reformatted; we’ve added line breaks and reworked indentation to accommodate the available page width in the printed book. In some cases, even this was not enough, and listings include line-continuation markers (➥). Additionally, code annotations accompany many of the listings, highlighting important concepts.
All the code samples in this book are available on GitHub at https://github.com/vladris/azure-data-engineering. The code was thoroughly tested, but because the Azure cloud and surrounding tooling continuously evolves, check appendix C if you run into issues trying any of the code samples.
liveBook discussion forum
Purchase of Data Engineering on Azure includes free access to a private web forum run by Manning Publications where you can make comments about the book, ask technical questions, and receive help from the author and from other users. To access the forum, go to https://livebook.manning.com/#!/book/data-engineering-on-azure/discussion. You can also learn more about Manning’s forums and the rules of conduct at https://livebook.manning.com/#!/discussion.
Manning’s commitment to our readers is to provide a venue where a meaningful dialogue between individual readers and between readers and the author can take place. It is not a commitment to any specific amount of participation on the part of the author, whose contribution to the forum remains voluntary (and unpaid). We suggest you try asking him some challenging questions lest his interest stray! The forum and the archives of previous discussions will be accessible from the publisher’s website as long as the book is in print.
about the author
Vlad Riscutia is a software engineer at Microsoft, where he oversees development of the data platform supporting the central data science team for Azure. He spent the past few years as an architect on the Customer Growth and Analytics team, building out a big data platform used by Azure’s data science organization. He has headed up several major software projects and mentors up-and-coming software engineers.
about the cover illustration
The figure on the cover of Data Engineering on Azure is captioned Femme Tartar,
or Tartar woman. The illustration is taken from a collection of dress costumes from various countries by Jacques Grasset de Saint-Sauveur (1757–1810), titled Costumes de Différents Pays, published in France in 1797. Each illustration is finely drawn and colored by hand. The rich variety of Grasset de Saint-Sauveur’s collection reminds us vividly of how culturally apart the world’s towns and regions were just 200 years ago. Isolated from each other, people spoke different dialects and languages. In the streets or in the countryside, it was easy to identify where they lived and what their trade or station in life was just by their dress.
The way we dress has changed since then and the diversity by region, so rich at the time, has faded away. It is now hard to tell apart the inhabitants of different continents, let alone different towns, regions, or countries. Perhaps we have traded cultural diversity for a more varied personal life—certainly for a more varied and fast-paced technological life.
At a time when it is hard to tell one computer book from another, Manning celebrates the inventiveness and initiative of the computer business with book covers based on the rich diversity of regional life of two centuries ago, brought back to life by Grasset de Saint-Sauveur’s pictures.
1 Introduction
This chapter covers
Defining data engineering
Anatomy of a data platform
Benefits of the cloud
Getting started with Azure
Overview of an Azure data platform
With the advent of cloud computing, the amount of data generated every moment reached an unprecedented scale. The discipline of data science flourishes in this environment, deriving knowledge and insights from massive amounts of data. As data science becomes critical to business, its processes must be treated with the same rigor as other components of business IT. For example, software engineering teams today embrace DevOps to develop and operate services with 99.99999% availability guarantees. Data engineering brings a similar rigor to data science, so data-centric processes run reliably, smoothly, and in a compliant way.
For the past few years, I’ve had the privilege of being a software architect for Microsoft’s Customer Growth and Analytics team. Our team’s motto is Using Azure to understand Azure.
We connect many datapoints across the Microsoft business to better understand our customers and to empower teams across the company. Privacy is important to us, so we never look at our customers’ data, but we do have access to telemetry from Azure, commercial transactions, and other operational pipelines. This gives us a unique perspective on Azure in understanding how customers can get the most value from our offerings.
As a few examples, we help marketing, sales, support, finance, operations, and business planning with key insights, while simultaneously providing operational excellence recommendations to our customers through Azure Advisor. While our data science and machine learning (ML) teams focus on the insights, our data engineering teams ensure we can operate at the scale of an Azure business with high reliability because any outage in our platform can impact our customers or our business.
Our data platform is fully built on Azure, and we are working closely with service teams to preview features and give product feedback. This book is inspired by some of our learnings over the years. The technologies presented are close to what my team uses on a day-to-day basis.
1.1 What is data engineering?
This book is about practical data engineering in a production environment, so let’s start by defining data engineering. But to define data engineering, we first need to talk about data science.
Data is the new oil,
as the saying goes. In a connected world, more and more data is available for analysis, inference, and ML. The field of data science deals with extracting knowledge and insights from data. Many times, these insights prove invaluable to a business. Consider a scenario like the movies Netflix recommends to a customer. The better the recommendations, the more likely to retain a customer.
While many data science projects start as exploratory, once these show real value, they need to be supported in an ongoing, reliable fashion. In the software engineering world, this is the equivalent of taking a research, proof-of-concept, or hackathon project and graduating it into a fully production-ready solution. While a hack or a prototype can take many shortcuts and focus on the meat of the problem
it addresses, a production-ready system does not cut any corners. This is where the engineering part of software engineering comes into play, providing the rigor to build and run a reliable system. This includes a plethora of concerns like architecture and design, performance, security, accessibility, telemetry, debuggability, extensibility, and so on.
Definition Data engineering is the part of data science that deals with the practical applications of collecting and analyzing data. It aims to bring engineering rigor to the process of building and supporting reliable data systems.
The ML part of data science deals with building a model. In the Netflix scenario, the data model recommends, based on your viewing history, which movies you are likely to enjoy next. The data engineering part of the discipline deals with building a system that continuously gathers and cleans up the viewing history, then runs the model at scale on the data of all users and distributes the results to the recommendation user interface. All of this provided in an automated fashion with monitoring and alerting build around each step of the process.
Data engineering deals with building and operating big data platforms to support all data science scenarios. There are various other terms used for some of these aspects: DataOps refers to moving data in a data system, MLOps refers to running ML at scale as in our Netflix example. (ML combined with DevOps is also known as MLOps.) Our definition of data engineering encompasses all of these and looks at how we can implement DevOps for data science.
1.2 Who this book is for
This is a book for data scientists, software engineers, and software architects turned data engineers and tasked with building a data platform to support analytics and/or ML at scale. You should know what the cloud is, have some experience working with data and code, and not mind using a shell. We’ll touch on the basics of all of these, but the focus for this book will be on data platform building.
Data engineering is surprisingly similar to software engineering and frustratingly different. While we can leverage a lot of the lessons from the software engineering world, as we will see in this book, there is a unique set of challenges we will have to address. Some of the common themes are making sure everything is tracked in source control, automatic deployments, monitoring, and alerting. A key difference between data and code is that code is static: once the bugs are worked out, a piece of code is expected to work consistently and reliably. On the other hand, data moves continuously into and out of a data platform, and it is likely failures will occur due to various external reasons. Governance is another major topic that is specific to data: access control, cataloguing, privacy, and regulatory concerns are a big part of a data platform.
The main theme of the book is bringing some of the lessons learned from software engineering over the past few decades to the data space so you can build a data platform exhibiting the properties of a solid software solution: scale, reliability, security, and so on. This book tackles some of these challenges, goes over patterns and best practices, and provides examples of how these could be applied in the Azure cloud. For the examples, we will use the Azure CLI (CLI stands for command-line interface), KQL (the Kusto Query Language), and a little bit of Python. The focus won’t be on the services themselves though. Instead, we will focus on data engineering challenges (and solutions) in a production environment.
1.3 What is a data platform?
Just as many data science projects start as an exploration of a data space and what insights can be derived from the data, many data science teams start in a similar exploratory fashion. A small team comes up with some good insights at first, and then as the team grows, so do the needs of the underlying platform supporting the team.
What first used to be an ad hoc process now requires automation. Once there were just two data scientists on the team, so who got to see which data was not as much of a concern as it is now, when there are 100 data scientists, some interns, and some external vendors. What used to be a monthly email is now a live system integrated with the company’s website. Different scenarios that used to be achieved through different means must now be supported by a robust data platform.
definition A data platform is a software solution for collecting, processing, managing, and sharing data for strategic business purposes.
Let’s look at an analogy to software engineering. You can write code on your laptop (for example, a web service like GIPHY) that, when given some keywords, returns a set of topical animations. Even if the code does exactly what it is meant to, that doesn’t mean it can scale to a production environment. If you want to host the same service at web scale and expect that anyone around the world can access it at any time, there is an additional set of concerns to consider: performance, scaling to millions of users, low latency, a failover solution in case things go wrong, a way to deploy an update without downtime, and so on. We can call the first part, writing code on your laptop, software development or coding. The second part, operating a production service, we can call software engineering.
The same applies to data engineering. Running a data platform at scale comes with a unique set of challenges to consider and address. Data science deals with writing queries and developing ML models. Data engineering takes these and scales them to millions of rows of data, provides automation and monitoring, ensures security and compliance, and so on. These aspects are the main focus of this book.
1.3.1 Anatomy of a data platform
The data platform grows to support all these new production scenarios, converting ad hoc processing into automated workflows and applying best practices. At this scale, certain patterns emerge. Figure 1.1 shows the anatomy of such a platform. Because we are dealing with data, many of the visuals focus on data flows.
Figure 1.1 On the left, data is ingested into the system and persisted in a storage layer. Processing aggregates and reshapes the data to enable analytics and ML scenarios. Orchestration and governance are cross-cutting concerns that cover all the components of the platform. Once processed, data is distributed to other downstream systems. All components are tracked by and deployed from source control.
Part 1 of the book focuses on infrastructure, the core services of a data platform. These include storage and analytics services, automatic deployment and monitoring, and an orchestration solution.
We’ll start with storage—the backbone of any data platform. Chapter 2 covers the requirements and common patterns for storing data in a data platform. Because our focus is on production systems, in chapter 3, we’ll discuss DevOps and what DevOps means for data. Data is ingested into the system from multiple sources. Data flows into and out of the platform, and various workflows are executed. All of this needs an orchestration layer to keep things running. We’ll talk about orchestration in chapter 4.
Part 2 focuses on the three main workloads that a data platform must support. These are
Processing—Encompasses aggregating and reshaping the data, standardizing schema, and any other processing of the raw input data. This makes the data easier to consume by the other two main processes: analytics and machine learning. We’ll talk about data processing in chapter 5.
Analytics—Covers all data analysis and reporting, thereby deriving knowledge and insights on the data. We’ll look at ways to support this in production in chapter 6.
Machine learning—Includes all ML models training on the data. We’ll cover running ML at scale in chapter 7.
Part 3 covers governance, a major topic with many aspects. Chapters 8, 9, and 10 touch on these key topics:
Metadata—Cataloguing and inventorying the data, tracking lineage, definitions, and documentation is the subject of chapter 8.
Data quality—How to test data and assess its quality is the topic of chapter 9.
Compliance—Honoring compliance requirements like the General Data Protection Regulation (GDPR), handling sensitive data, and controlling access is covered in chapter 10.
After all the processing steps, data eventually leaves the platform to be consumed by other systems. We’ll cover the various patterns for distributing data in chapter 11. Data governance is a pretty loose term, so let’s work with the following definition:
Definition Governance is the process of managing the availability, usability, integrity, regulatory compliance, and security of the data in a data system. Effective data governance ensures that data is consistent and trustworthy and doesn't get misused.
On one hand, governance is needed to reduce liability, making sure data complies with regulations, is secure, and so on. On the other hand, governance also includes making data discoverable, ensuring it is high-quality and, in general, increasing the usability of the platform.
Infrastructure-wise, the topics discussed apply to any data platform, regardless of whether it is implemented on premises, in the Azure cloud, in AWS (Amazon Web Services), and so on. We need to work with some concrete examples, though, so this book covers the implementation of a data platform in the Azure cloud.
Even within Azure, there are multiple services that support analytics, ML, and so on. For example, we can use Azure Databricks, Azure Machine Learning (AML), or Azure HDInsight/Spark to train ML models, and we can use Azure Synapse, Azure Data Explorer (ADX), or Azure Databricks to perform analytics. This book covers one possible implementation but, as every software architect knows, there are always trade-offs. Depending on your scenario, you might pick a different technology to implement your data platform. There is no right way.
Many factors inform the technology choice: existing assets, what the users of the platform are familiar with, portability, performance for various workloads, and so on. We will look at some of these key differences and zoom in on one possible implementation. As you read, keep in mind that the underlying patterns are more important than the particular technology choice, and you might choose to materialize these on a different technology stack.
1.3.2 Infrastructure as code, codeless infrastructure
Because we are dealing with production systems, we’ll focus a lot on DevOps and best practices. This includes avoiding interactive configuration tools and automating everything via scripts and machine-readable configurations, also known as infrastructure as code.
definition Infrastructure as code is the process of managing and provisioning infrastructure through automation by relying on configuration files and automation scripts as opposed to manual and interactive configurations.
Surprisingly, focusing on infrastructure as code doesn’t mean we will have to write thousands of lines of code to build a data platform. In fact, most of the components we need are readily available and only need to be configured and stitched together to support our scenarios. Such an infrastructure using mostly off-the-shelf components and a little glue is called a codeless infrastructure.
definition Codeless infrastructure is an infrastructure built by configuring existing services and connecting them to achieve the required scenarios. This is done with as little custom code as possible.
In general, code is not an asset; rather, it is a liability. What the code does, the scenarios it enables, is the real asset. The code itself needs maintenance, has bugs, requires updates, and in general, consumes engineering time and resources. When possible, it’s better to let others worry about this maintenance. Today, most of the infrastructure we need is offered as services by cloud providers like Microsoft and Amazon. We will use Azure, Microsoft’s cloud offering, to implement the examples in this book.
With these services, a small engineering team can achieve a (surprising) lot. Focus moves from developing infrastructure to configuring, deploying, and monitoring it, and then focusing on solving some of the higher-level challenges of the domain. In our case, these challenges are around scaling out data workloads and governance concerns.
1.4 Building in the cloud
Big data comes from operating at scale. The amount of data grows with the number of people and devices connected to the internet and the information these generate. As infrastructure becomes commoditized in the cloud, data platforms are built in the cloud too. We used to run analytics on SQL Servers hosted on premises with over hundreds of megabytes, maybe even gigabytes, of data. Now we can run analytics on hundreds of gigabytes or even terabytes of data in the cloud, using specialized storage and distributed querying solutions. We can rent these solutions from multiple cloud providers like Microsoft, Amazon, or Google.
1.4.1 IaaS, PaaS, SaaS
Cloudsolutions are usually categorized into infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). IaaS provides virtualized computing resources like networking, storage, and virtual machines (VMs). Instead of buying computers, networking equipment, and ensuring that these are properly set up and running, we can rent them from a cloud provider. If we suddenly need more capacity, we can easily request more. If we need less capacity, we can free that up almost instantly. This ends up being much cheaper than building and maintaining a small data center. But it doesn’t stop here.
PaaS provides higher-level abstractions than just the basic computing resources. Instead of renting infrastructure on which we install an SQL Server, we can rent a fully managed Azure SQL instance. This is a database handled by Azure that includes high availability, automatic installation of software updates, threat detection, and many other features