Engineering MLOps: Rapidly build, test, and manage production-ready machine learning life cycles at scale

BIRMINGHAM—MUMBAI

Engineering MLOps

Copyright © 2021 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Kunal Parikh

Publishing Product Manager: Aditi Gour

Senior Editor: Mohammed Yusuf Imaratwale

Content Development Editor: Nazia Shaikh

Technical Editor: Arjun Varma

Copy Editor: Safis Editing

Project Coordinator: Aishwarya Mohan

Proofreader: Safis Editing

Indexer: Priyanka Dhadke

Production Designer: Joshua Misquitta

First published: April 2021

Production reference: 2261021

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80056-288-2

www.packt.com

Contributors

About the author

Emmanuel Raj is a Finland-based Senior Machine Learning Engineer with 6+ years of industry experience. He is also a Machine Learning Engineer at TietoEvry and a Member of the European AI Alliance at the European Commission. He is passionate about democratizing AI and bringing research and academia to industry. He holds a Master of Engineering degree in Big Data Analytics from Arcada University of Applied Sciences. He has a keen interest in R&D in technologies such as Edge AI, Blockchain, NLP, MLOps, and Robotics. He believes the best way to learn is to teach, he is passionate about sharing and learning new technologies with others.

About the reviewers

Magnus Westerlund (DSc) is a principal lecturer in information technology and director of the master's degree programme in big data analytics at Arcada University of Applied Sciences in Helsinki, Finland. He has a background in telecoms and information management and earned his doctoral degree in information systems at Åbo Akademi University, Finland. Magnus has published research in the fields of analytics, IT security, cyber regulation, and distributed ledger technology. His current research topics are smart contract-based distributed security for IoT edge applications and the assessment of intelligent systems. He participates as a technical expert in the Z-inspection® network, which works for a Mindful Use of AI (#MUAI).

Stephen Oladele is the co-founder of AgServer, a peer-to-peer knowledge-sharing platform for smallholder farmers in Africa. He also assists in building data science talents at TheGradientBoost, a staffing and recruiting company. He has consulted as a data scientist for companies and individuals, helping them go from business ideas to execution with notable projects in computer vision, business analytics, and NLP (document analysis), using cloud machine learning services such as those in Google Cloud Platform and Microsoft Azure. In his spare time, he loves volunteering. He runs nonprofit organizations helping underrepresented groups in Africa get into AI and technology. He has volunteered with Google and AWS for projects in the past.

Emerson Bertolo is a data scientist and software developer who has created mission-critical software and dealt with big data applications for more than 12 years. In 2016, Bertolo deep-dived into machine learning and deep learning projects by creating AI models using TensorFlow, PyTorch, MXNet, Keras, and Python libraries to bring those models into reality for tech companies from LawTech to security and defense. By merging Agile concepts into data science, Bertolo has been seeking the best blend between Agile software engineering and machine learning research to build time-to-market AI applications. His approach has been build to learn, validate results, research and identify uncertainties, rebuild, and learn again!

Table of Contents

Preface

Section 1: Framework for Building Machine Learning Models

Chapter 1: Fundamentals of an MLOps Workflow

The evolution of infrastructure and software development

The rise of machine learning and deep learning5

The end of Moore's law7

AI-centric applications7

Software development evolution8

Traditional software development challenges

Trends of ML adoption in software development

Understanding MLOps

Concepts and workflow of MLOps

Discussing a use case 17

Summary

Chapter 2: Characterizing Your Machine Learning Problem

The ML solution development process

Types of ML models

Learning models30

Hybrid models31

Statistical models 34

HITL models36

Structuring your MLOps

Small data ops39

Big data ops40

Hybrid MLOps41

Large-scale MLOps41

An implementation roadmap for your solution

Phase 1 – ML development43

Phase 2 – Transition to operations44

Phase 3 – Operations44

Procuring data, requirements, and tools

Data45

Requirements46

Tools and infrastructure

Discussing a real-life business problem

Summary

Chapter 3: Code Meets Data

Business problem analysis and categorizing the problem

Setting up the resources and tools

Installing MLflow54

Azure Machine Learning55

Azure DevOps58

JupyterHub60

10 principles of source code management for ML

What is good data for ML?

Data preprocessing

Data quality assessment66

Calibrating missing data68

Label encoding 68

New feature – Future_weather_condition70

Data correlations and filtering70

Time series analysis73

Data registration and versioning

Toward the ML Pipeline

Feature Store 77

Summary

Chapter 4: Machine Learning Pipelines

Going through the basics of ML pipelines

Data ingestion and feature engineering

Data ingestion (training dataset)90

Machine learning training and hyperparameter optimization

Support Vector Machine92

Random Forest classifier94

Model testing and defining metrics

Testing the SVM classifier96

Testing the Random Forest classifier97

Model packaging

Registering models and production artifacts

Registering production artifacts100

Summary

Chapter 5: Model Evaluation and Packaging

Model evaluation and interpretability metrics

Learning models' metrics105

Hybrid models' metrics114

Statistical models' metrics119

HITL model metrics121

Production testing methods

Batch testing123

A/B testing123

Stage test or shadow test124

Testing in CI/CD124

Why package ML models?

Portability 125

Inference126

Interoperability 126

Deployment agnosticity 126

How to package ML models

Serialized files127

Packetizing or containerizing 128

Microservice generation and deployment129

Inference ready models

Connecting to the workspace and importing model artifacts131

Loading model artifacts for inference131

Summary

Section 2: Deploying Machine Learning Models at Scale

Chapter 6: Key Principles for Deploying Your ML System

ML in research versus production

Data136

Fairness137

Interpretability138

Performance138

Priority 139

Understanding the types of ML inference in production

Deployment targets139

Mapping the infrastructure for our solution145

Hands-on deployment (for the business problem)

Deploying the model on ACI146

Deploying the model on Azure Kubernetes Service (AKS)153

Deploying the service using MLflow 161

Understanding the need for continuous integration and continuous deployment

Summary

Chapter 7: Building Robust CI/CD Pipelines

Continuous integration, delivery, and deployment in MLOps

Continuous integration167

Continuous delivery167

Continuous deployment168

Setting up a CI/CD pipeline and the test environment (using Azure DevOps)

Creating a service principal169

Installing the extension to connect to the Azure ML workspace172

Setting up a continuous integration and deployment pipeline for the test environment174

Connecting artifacts to the pipeline175

Setting up a test environment178

Pipeline execution and testing

Pipeline execution triggers

Summary

Chapter 8: APIs and Microservice Management

Introduction to APIs and microservices

What is an Application Programming Interface (API)?192

Microservices193

The need for microservices for ML

Hypothetical use case195

Stage 1 – Proof of concept (a monolith)195

Stage 2 – Production (microservices)196

Old is gold – REST API-based microservices

Hands-on implementation of serving an ML model as an API

API design and development 200

Developing a microservice using Docker

Testing the API

Summary

Chapter 9: Testing and Securing Your ML Solution

Understanding the need for testing and securing your ML application

Testing your ML solution by design

Data testing214

Model testing215

Pre-training tests216

Post-training tests216

Hands-on deployment and inference testing (a business use case)

Securing your ML solution by design

Types of attacks225

Summary

Chapter 10: Essentials of Production Release

Setting up the production infrastructure

Azure Machine Learning workspace231

Azure Machine Learning SDK234

Setting up our production environment in the CI/CD pipeline

Testing our production-ready pipeline

Configuring pipeline triggers for automation

Setting up a Git trigger 246

Setting up an Artifactory trigger247

Setting up a Schedule trigger248

Pipeline release management

Toward continuous monitoring

Summary

Section 3: Monitoring Machine Learning Models in Production

Chapter 11: Key Principles for Monitoring Your ML System

Understanding the key principles of monitoring an ML system

Model drift257

Model bias 258

Model transparency 258

Model compliance 259

Explainable AI 260

Monitoring in the MLOps workflow

Understanding the Explainable Monitoring Framework

Monitor 265

Analyze 267

Govern271

Enabling continuous monitoring for the service

Summary

Chapter 12: Model Serving and Monitoring

Serving, monitoring, and maintaining models in production

Exploring different modes of serving ML models

Serving the model as a batch service278

Serving the model to a human user279

Serving the model to a machine 280

Implementing the Explainable Monitoring framework

Monitoring your ML system284

Analyzing your ML system307

Governing your ML system

Summary

Chapter 13: Governing the ML System for Continual Learning

Understanding the need for continual learning

Continual learning 312

The need for continual learning 313

Explainable monitoring – governance

Alerts and actions315

Model QA and control333

Model auditing and reports 333

Enabling model retraining

Manual model retraining 336

Automated model retraining 336

Maintaining the CI/CD pipeline

Summary

Other Books You May Enjoy

Preface

MLOps is a systematic approach to building, deploying, and monitoring machine learning (ML) solutions. It is an engineering discipline that can be applied to various industries and use cases. This book presents comprehensive insights into MLOps coupled with real-world examples to help you to write programs, train robust and scalable ML models, and build ML pipelines to train and deploy models securely in production.

You will begin by familiarizing yourself with MLOps workflow and start writing programs to train ML models. You’ll then move on to explore options for serializing and packaging ML models post-training to deploy them in production to facilitate machine learning inference. Next, you will learn about monitoring ML models and system performance using an explainable monitoring framework. Finally, you’ll apply the knowledge you’ve gained to build real-world projects.

By the end of this ML book, you'll have a 360-degree view of MLOps and be ready to implement MLOps in your organization.

Who this book is for

This MLOps book is for data scientists, software engineers, DevOps engineers, machine learning engineers, and business and technology leaders who want to build, deploy, and maintain ML systems in production using MLOps principles and techniques. Basic knowledge of machine learning is necessary to get started with this book.

What this book covers

Chapter 1, Fundamentals of MLOps Workflow, gives an overview of the changing software development landscape by highlighting how traditional software development is changing to facilitate machine learning. We will highlight some daily problems within organizations with the traditional approach, showcasing why a change in thinking and implementation is needed. Proceeding that an introduction to the importance of systematic machine learning will be given, followed by some concepts of machine learning and DevOps and fusing them into MLOps. The chapter ends with a proposal for a generic workflow to approach almost any machine learning problem. 

Chapter 2, Characterizing Your Machine Learning Problem, offers you a broad perspective on possible types of ML solutions for production. You will learn how to categorize solutions, create a roadmap for developing and deploying a solution, and procure the necessary data, tools, or infrastructure to get started with developing an ML solution taking a systematic approach. 

Chapter 3, Code Meets Data, starts the implementation of our hands-on business use case of developing a machine learning solution. We discuss effective methods of source code management for machine learning, data processing for the business use case, and formulate a data governance strategy and pipeline for machine learning training and deployment.

Chapter 4, Machine Learning Pipelines, takes a deep dive into building machine learning pipelines for solutions. We look into key aspects of feature engineering, algorithm selection, hyperparameter optimization, and other aspects of a robust machine learning pipeline.

Chapter 5, Model Evaluation and Packaging, takes a deep dive into options for serializing and packaging machine learning models post-training to deploy them at runtime to facilitate machine learning inference, model interoperability, and end-to-end model traceability. You'll get a broad perspective on the options available and state-of-the-art developments to package and serve machine learning models to production for efficient, robust, and scalable services. 

Chapter 6, Key Principles for Deploying Your ML System, introduces the concepts of continuous integration and deployment in production for various settings. You will learn how to choose the right options, tools, and infrastructure to facilitate the deployment of a machine learning solution. You will get insights into machine learning inference options and deployment targets, and get an introduction to CI/CD pipelines for machine learning. 

Chapter 7, Building Robust CI and CD Pipelines, covers different CI/CD pipeline components such as triggers, releases, jobs, and so on. It will also equip you with knowledge on curating your own custom CI/CD pipelines for ML solutions. We will build a CI/CD pipeline for an ML solution for a business use case. The pipelines we build will be traceable end to end as they will serve as middleware for model deployment and monitoring.

Chapter 8, APIs and Microservice Management, goes into the principles of API and microservice design for ML inference. A learn by doing approach will be encouraged. We will go through a hands-on implementation of designing and developing an API and microservice for an ML model using tools such as FastAPI and Docker. You will learn key principles, challenges, and tips to designing a robust and scalable microservice and API for test and production environments.

Chapter 9, Testing and Securing Your ML Solution, introduces the core principles of performing tests in the test environment to test the robustness and scalability of the microservice or API we have previously developed. We will perform hands-on load testing for a deployed ML solution. This chapter provides a checklist of tests to be done before taking the microservice to production release.

Chapter 10, Essentials of Production Release, explains how to deploy ML services to production with a robust and scalable approach using the CI/CD pipelines designed earlier. We will focus on deploying, monitoring, and managing the service in production. Key learnings will be deployment in serverless and server environments using tools such as Python, Docker, and Kubernetes.

Chapter 11, Key Principles for Monitoring Your ML System, looks at key principles and aspects of monitoring ML systems in production for robust, secure, and scalable performance. As a key takeaway, readers will get a concrete explainable monitoring framework and checklist to set up and configure a monitoring framework for their ML solution in production. 

Chapter 12, Model Serving and Monitoring, explains serving models to users and defining metrics for an ML solution, especially in the aspects of algorithm efficiency, accuracy, and production performance. We will deep dive into hands-on implementation and real-life examples on monitoring data drift, model drift, and application performance.

Chapter 13, Governing the ML System for Continual Learning, reflects on the need for continual learning in machine learning solutions. We will look into what is needed to successfully govern an ML system for business efficacy. Using the Explainable Monitoring framework, we will devise a strategy to govern and we will delve into the hands-on implementation for error handling and configuring alerts and actions. This chapter will equip you with critical skills to automate and govern your MLOps.

To get the most out of this book

You should have access to a Microsoft Azure subscription and basic DevOps-based software used to build CI/CD pipelines. A personal computer or laptop with a Linux or macOS is a plus.

If you are using the digital version of this book, we advise you to type the code yourself or access the code via the GitHub repository (link available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/EngineeringMLOps. In case there's an update to the code, it will be updated on the existing GitHub repository. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781800562882_ColorImages.pdf.

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: The preprocessed dataset is imported using the .get_by_name() function.

A block of code is set as follows:

uri = workspace.get_mlflow_tracking_uri( )

mlflow.set_tracking_uri(uri)

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

# Importing pre-processed dataset

dataset = Dataset.get_by_name (workspace, name='processed_weather_data_portofTurku')

Any command-line input or output is written as follows:

python3 test_inference.py

Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: Go to the Compute option and click the Create button to explore compute options available on the cloud.

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at [email protected].

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details.

Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Reviews

Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you!

For more information about Packt, please visit packt.com.

Section 1: Framework for Building Machine Learning Models

This part will equip readers with the foundation of MLOps and workflows to characterize their ML problems to provide a clear roadmap for building robust and scalable ML pipelines. This will be done in a learn-by-doing approach via practical implementation using proposed methods and tools (Azure Machine Learning services or MLflow).

This section comprises the following chapters:

Chapter 1, Fundamentals of MLOps WorkFlow

Chapter 2, Characterizing Your Machine Learning Problem

Chapter 3, Code Meets Data

Chapter 4, Machine Learning Pipelines

Chapter 5, Model Evaluation and Packaging

Chapter 1: Fundamentals of an MLOps Workflow

Machine learning (ML) is maturing from research to applied business solutions. However, the grim reality is that only 2% of companies using ML have successfully deployed a model in production to enhance their business processes, reported by DeepLearning.AI (https://info.deeplearning.ai/the-batch-companies-slipping-on-ai-goals-self-training-for-better-vision-muppets-and-models-china-vs-us-only-the-best-examples-proliferating-patents). What makes it so hard? And what do we need to do to improve the situation?

To get a solid understanding of this problem and its solution, in this chapter, we will delve into the evolution and intersection of software development and ML. We'll begin by reflecting on some of the trends in traditional software development, starting from the waterfall model to agile to DevOps practices, and how these are evolving to industrialize ML-centric applications. You will be introduced to a systematic approach to operationalizing AI using Machine Learning Operations (MLOps). By the end of this chapter, you will have a solid understanding of MLOps and you will be equipped to implement a generic MLOps workflow that can be used to build, deploy, and monitor a wide range of ML applications.

In this chapter, we're going to cover the following main topics:

The evolution of infrastructure and software development

Traditional software development challenges

Trends of ML adoption in software development

Understanding MLOps

Concepts and workflow of MLOps

The evolution of infrastructure and software development

With the genesis of the modern internet age (around 1995), we witnessed a rise in software applications, ranging from operating systems such as Windows 95 to the Linux operating system and websites such as Google and Amazon, which have been serving the world (online) for over two decades. This has resulted in a culture of continuously improving services by collecting, storing, and processing a massive amount of data from user interactions. Such developments have been shaping the evolution of IT infrastructure and software development.

Transformation in IT infrastructure has picked up pace since the start of this millennium. Since then, businesses have increasingly adopted cloud computing as it opens up new possibilities for businesses to outsource IT infrastructure maintenance while provisioning necessary IT resources such as storage and computation resources and services required to run and scale their operations.

Cloud computing offers on-demand provisioning and the availability of IT resources such as data storage and computing resources without the need for active management by the user of the IT resources. For example, businesses provisioning computation and storage resources do not have to manage these resources directly and are not responsible for keeping them running – the maintenance is outsourced to the cloud service provider.

Businesses using cloud computing can reap benefits as there's no need to buy and maintain IT resources; it enables them to have less in-house expertise for IT resource maintenance and this allows businesses to optimize costs and resources. Cloud computing enables scaling on demand and users pay as per the usage of resources. As a result, we have seen companies adopting cloud computing as part of their businesses and IT infrastructures.

Cloud computing became popular in the industry from 2006 onward when Sun Microsystems launched Sun Grid in March 2006. It is a hardware and data resource sharing service. This service was acquired by Oracle and was later named Sun Cloud. Parallelly, in the same year (2006), another cloud computing service was launched by Amazon called Elastic Compute Cloud. This enabled new possibilities for businesses to provision computation, storage, and scaling capabilities on demand. Since then, the transformation across industries has been organic toward adopting cloud computing.

In the last decade, many companies on a global and regional scale have catalyzed the cloud transformation, with companies such as Google, IBM, Microsoft, UpCloud, Alibaba, and others heavily investing in the research and development of cloud services. As a result, a shift from localized computing (companies having their own servers and data centers) to on-demand computing has taken place due to the availability of robust and scalable cloud services. Now businesses and organizations are able to provision resources on-demand on the cloud to satisfy their data processing needs.

With these developments, we