Mlops Ebook With Preview
Mlops Ebook With Preview
Practical
MLOPS HOW TO GET READY FOR
PRODUCTION MODELS
Contents
Why MLOps Matters? (Foreword) 3
Conclusion 50
50%
40%
30%
20%
10%
0%
Collect more Figure out how Make data more Learn machine Prove potential Develop models Deploy models Optimize Automate Monitor
data data can be accessible learning of machine for production to production models retraining model(s) in
used technologies learning use model(s) production
4 Why MLOps Matters?
Remember that these roles are not necessarily one per person but
rather a single person can - and in smaller organizations often has to -
cover multiple roles. For example, many data scientists also handle ML
Engineering tasks, and sometimes DevOps and IT are synonymous.
The rule of thumb is the larger the organization, the more specialized
and siloed individuals are which requires all the more collaboration.
Data Scientist
Data scientists are tasked with finding data-driven solutions to
business problems. Data scientists tend to be highly educated with
a master’s degree or higher in mathematics and statistics, computer
science or engineering. Their strengths lie in data wrangling, statistical
analysis, data visualization and applying machine learning.
Generally, they are deployed first to analyze existing data and to find
6 People of Machine Learning
Data Engineer
Data engineers are responsible for the infrastructure that ensures
data is stored and available for data scientists. They build the systems
that ingest raw data from various sources and store it in a centralized
data lake. Data engineers often also build other data warehouses
that derive data from the data lake but transform it so that it is more
useful for the end-user; for example, image data might be resized and
reformatted so that data scientists can focus on data analysis rather
than data manipulation.
DevOps Engineer
DevOps engineers combine the domains of software engineering
and cloud infrastructure. In traditional software projects, DevOps
People of Machine Learning 7
IT
IT operations can be found in most larger organizations. They are
generally tasked with resource management, access control and
information security. While often seen as red tape, IT operations and
the processes involved can help a project run more smoothly, whether
it’s through requesting that computing resources be provisioned or
ensuring that outside vendors are approved.
Business Owner
There isn’t a specific title for a business owner in a machine learning
environment but it’s a key role to ensure success. Business owners
have the deepest understanding of the end-user and use case, and
guide the team to producing ML systems that are not just accurate
and well-engineered but also valuable. They’ll help define what kind
of predictions are useful and help understanding of where risks lie
from an operational and regulatory perspective.
Manager
The management role in a team working with machine learning tends
to be a little unclear as these teams tend to be relatively new. The key
aspects to consider are that machine learning projects are deceptively
complex and multidisciplinary. All the required resources to execute
a project are often not in a single team and a manager must secure
these from different parts of the organization. Another difficult task
for a manager is to understand and communicate the ROI of the team’s
work as costs and benefits are likely not directly linked with the team.
8 How Is Machine Learning Different from Traditional Software?
However, it’s not only how the introduction of data changes how code
works in the development process but also that data is another aspect
that can change. In classical software development, a version of the
code produces a version of the software, while in machine learning,
a version of the code and a version of the data together produce a
version of the ML model.
How Is Machine Learning Different from Traditional Software? 9
Model
Design Operations
Development
As fun as development in a vacuum is (we all have done it), the reality
is that only a model running in production can bring value. Models
have zero ROI until they can be used. Therefore, time to market should
be the number one metric to look at and optimize for any ML project.
The MLOps Workflow 11
Time to Market
10% Model
Development
Design Operations
90% Glue
Coding
Complex &
unstructured
handover
Let’s break down the statement. It contains two parts, reducing the
time to market and reducing the risk involved. These goals help data
scientists and business stakeholders to speak the same language and
frame MLOps as a business-driven necessity. Albeit MLOps will help
automate manual steps and improve the quality of code, but they are
not the underlying goals the whole organization will rally around.
Next, we will talk about risk and time to market. The area of risk
assessment is a more straightforward concept, so let’s start there.
1. Loss of knowledge
2. Failures in production
3. Regulatory and ethical
12 The MLOps Workflow
Keep in mind that we are skipping basic IT risk that is already well
documented and covered elsewhere but is not exclusive to machine
learning in production; for example, data security. It’s clear that no-
one should be able to steal data from your database, but that is true
for all databases – not just ones used for machine learning.
Loss of Knowledge
Bus factor
tend to forget details about your own work too. It can be challenging
to return to a six-month-old project and retrain a model without
breaking things. Many readers may have existing models running that
they dread getting back to for fear of breaking something.
You need to know what data the code is using and how that data came
to be. There are often multiple scripts bringing together different
data sources and doing feature aggregation to build the dataset you
need. Putting together the puzzle of several scripts and data sources
someone else made leaving behind zero documentation can often be
more complicated than just starting from scratch.
Failures in Production
The next category of risk is failures in production. Everyone who has
worked in software development knows that shipping broken code
14 The MLOps Workflow
The same thing applies to ML. From basic syntax errors to changes in
the data stream or unexpected model performance.
In a perfect world, not only does your MLOps tooling support various
methods of testing your code and data throughout the pipeline, but
you’ve also adopted a practice to start codifying these tests, starting
from the design phase of an ML project.
1. Ensure that the data used to train a model looks like you expected
it to. There may be changes upstream, for example, how the data
is collected or stored.
2. Ensure that the model works not only in training but in a real-world
environment. Overfitting is a real risk that should be addressed by
your pipeline and process.
While MLOps can’t fix model biases, it comes with the notion that
these concerns should be addressed by codifying tests for them into
a machine learning pipeline. Checkpoints and safeguards for how the
data used for training should look and what the expected predictions
should be will make governance audits much less intimidating. All of
the production models will be produced in compliance with stable
and reliable rules.
Time to Market
It’s hard to beat an ad-hoc workflow on speed to market when first
developing a machine learning model. For many data scientists and
engineers, that is the reason they are hesitant to adopt rigor into their
process. However, machine learning in production is rarely about a
single version of a single model getting into production but rather a
long-term commitment to building machine learning capabilities that
16 The MLOps Workflow
Model
Customer 1 Customer 2
Time 1 Time 2 Time 3 Data Source 2 Time 1 Time 2 Time 3 Time 1 Time 2 Time 3 Data Source 2 Time 1 Time 2 Time 3
18 Trained Models
The MLOps Workflow 17
Broadly speaking, there are two ways that MLOps helps accelerate
the time to market of machine learning:
Shared Language
We have seen a drastic uptake in software development delivery speed
and rate over the years. Modern tooling and shared work methods
(CI/CD, version control, microservices) have enabled companies to
scale their throughput in software development exponentially.
A single data scientist doesn’t need to create the whole model, but
rather a team can work on separate pieces such as data preprocessing,
model training, and testing. As the complexity of model creation
increases, so do the benefits of working around a shared pipeline.
Automation
DevOps has allowed software developers to move from a monthly
18 The MLOps Workflow
Time to Market
Model
Design Operations
Development
Version Control
Pipelines
Testing
Automation
• Make sure you split your project into smaller pieces to validate
your assumptions, track your progress, and gather feedback.
The age-old “Push to ‘production’ often and iterate with
feedback” applies here as well. We’re not saying to skip long-
term engagement, but instead that you should show progress,
for example, every six weeks or so.
Taking this into production most likely won’t happen by copying the
model files over to the production server. In most organizations, the
path to production will be managed by the IT (or Ops) department,
which has been around for a while and has a mature infrastructure
with established practices. They will assume that you can automate
most (if not all) of your ML pipeline steps, from re-training the model
with new data, to comparing it to existing models, and publishing it to
production.
share the project’s objective and the individual metrics used to track
its success within the virtual team.
Assumptions
Success criteria
Each organization and each When defining the metrics for your
project will have it’s own custom success criteria, think about:
metrics.
• What is the key metric that will
You might focus on improving determine if the model can be taken
existing processes, or augmenting to production?
a manual process, or fully • How will the model be taken to
automating it. production?
• How often will the model need to
Whatever you’re doing, make sure be updated?
you’ve listed the outcomes (not • Model metrics (e.g. accuracy,
outputs) your virtual team should precision, recall)
track and what numbers should be • Cost to maintain and update the
reached. model (and pipeline)
• Ethical considerations
24 How to Quantify Success in an MLOps Project?
Success criteria
Company 1
Switch careers
Company 1 jumps right into the nitty-gritty and starts gathering and
analyzing data. Their lead data scientist develops a model in Jupyter
notebook on their laptop. The model is subsequently deployed to
production by a DevOps engineer from another team. From idea to
production, the delivery time is blazingly fast.
But trouble starts as time passes; user feedback shows that the
predictions the model produces are getting worse. Perhaps the
underlying data has changed, and the model needs to be retrained.
However, the first time around was poorly documented and done
mostly on a single computer. It’s hard to reproduce all the steps that it
took to train the model – but it gets done.
26 Real-World Example - The Story of Two Companies
Company 2
Select tooling Build pipeline Collect data Train model Evaluate model Deploy model
Automate
Learnings
These two stories illustrate why we don’t recommend considering ML
workflow as an afterthought. It is a fallacy to think, “We just aren’t
there yet.” The right time to consider tooling and workflow is when
the first data scientist is hired, not when the first model goes to
production or the prediction serving breaks for the nth time.
Real-World Example - The Story of Two Companies 27
Data Exploration
Without data, there can be no model, so data exploration precedes
model exploration quite naturally.
Model Exploration
Model exploration can overlap with the data exploration, but it can be
thought of as a separate step. During the model exploration step, the
data scientist explores the viability of different models like regression,
decision tree, or random forest to the problem at hand. Choosing the
model often requires the data scientist to try out and look for optimal
parameters (also known as hyperparameters) within the model.
Tools exist to find the optimal models and parameters automatically
(AutoML). Still, they tend not to work well for more complicated
projects and are often not viable at all for the deep learning cases.
Key Takeaways
1. Data exploration is a lightweight, offline task for understanding
data using visualization
Optimization Vocabulary
In software engineering (algorithms), optimizing a function means
rewriting it to be faster or smaller, which is an entirely different thing
and sometimes causes confusion.
Training/Validation/Test Datasets
In a production setting, it is common to divide all the available data
into three different datasets: training, validation and test datasets.
Dataset
Key Takeaways
1. Split available data into training, validation and test datasets.
Adapted from Google’s article, MLOps: Continuous delivery and automation pipelines in
machine learning
Offline data
ML Ops
Time goes by, and somebody alerts the team that the model might
be misbehaving. The original problem-solving capability seems to be
declining based on a gut feeling. Nobody knows for sure. The data
scientist manually grabs a new batch of offline data, starts the coffee
machine, and the entire manual cycle is ready to start all over again.
In the automated workflow, the data scientist doesn’t aim for a gigantic
notebook on her laptop with offline data. While she may initially start
with a single notebook, she will eventually split the problem-solving
into more manageable components.
The MLOps Toolchain - End-to-End Pipelines 37
Adapted from Google’s article, MLOps: Continuous delivery and automation pipelines in
machine learning
Automated pipeline
Trained
Data extraction Data validation Data preparation Model training Model evaluation Model validation
model
Model serving
Metadata store
The system offers the ability to execute, iterate, and monitor a single
component in the context of the entire pipeline with the same ease
and rapid iteration as running a local notebook cell on a laptop. It also
lets you define the required inputs and outputs, library dependencies,
and monitored metrics.
Key Takeaways
1. The pipeline is the product, not the model. Do not deploy the
model; deploy the pipeline.
For models, there are tools for building machine learning pipelines and
managing the model lifecycle. MLOps platforms allow data scientists
to train models, run experiments, validate models, and finally deploy
them to production.
For features, the tool offering has been much less mature. Getting
features to production is particularly hard because we need to get
feature transformations (or pipelines) to production, and curate
the feature values to serve consistent data for training and online
inference. Data scientists typically pass their feature transformations
to data engineers to re-implement the feature pipelines with
production-hardened code. That’s a complicated process that
typically introduces weeks or months of lead time.
40 The MLOps Toolchain - Feature Stores
Build Run
>_
APPS
DevOps Pipeline
Development Production
MODELS
MLOps Pipeline
Model Training Model Serving
FEATURES / DATA
Feature 1 Feature 1
Feature Store
Store
Serve
Transform
Data Scientists
In the past year, however, we’ve seen the introduction of several open
source and commercial feature stores. They integrate with existing
data lakes, data warehouses, event streaming platforms, processing
42 The MLOps Toolchain - Feature Stores
Key takeaways
1. Building features and getting them to production is one of the
hardest parts of productionizing ML.
Testing
Traditional software is built by writing fixed rules against well-
defined static assumptions of the surrounding world. It is relatively
straightforward to test every single rule (unit test) or group (integration
tests) when everything is defined in advance.
Data Testing
For an ML project, data is as (if not more) important than the code. Like
the unit tests for your code define and test your assumptions about
the inputs, your data validation tests should do the same for training
and inference input data. You should test for null values, abnormal
statistical distributions within a feature, and the relationships between
features.
is about products and has two similar columns for the price: taxes
included, and taxes excluded, it should trigger an alarm.
Model Testing
Once the assumptions about data are tested, we can move on to
test the models and their training. You shouldn’t just blindly deploy a
model that shows promising accuracy for your offline data and hope
for the best.
In the training phase, you can test the impact of each hyperparameter.
Running a grid-search or more advanced search strategy can uncover
reliability issues and also improve predictive performance. Using an
additional test set, disjoined from training and validation sets should
also be used whenever possible.
Another viable smoke test can be testing your new shiny model against
a simple baseline model. Trickle small amounts of live production data
to be handled by your new model as a canary test before you fully
commit.
Infrastructure Testing
Your ML pipeline should be as reproducible as possible from one day
to the next. Perfect determinism is hard, but you should work towards
it.
Key Takeaways
1. Due to the dynamic nature of ML, testing is even more critical.
The question one should always ask first is whether you aim for batch
inference or online inference; followed by whether to do classic
centralized cloud inference or distribute the compute requirements to
your customer’s hardware (edge inference).
Batch Inference
Database
98
Trained 95
model 87
81
74
Minutes to hours
Online Inference
Prediction service
Trained
API call API response
model
Milliseconds
Online inference suits any situation where the value provided by the
model is needed right away. For example, if the model is supposed to
predict the best route for an ambulance to take or the stock market’s
volatility for the next ten minutes, every second counts. The prediction
has no value if it’s late.
leaks back to the customer right away. The system also needs to be
automatically scalable to accommodate the peaks in traffic. Typically,
scaling is handled by using an auto-scalable cluster like Kubernetes.
Requirements for monitoring are much higher, and reaction time for
any intervention needs to be close to zero.
Edge Inference
On-device prediction
service
Trained Hello
model
Milliseconds
Another potential downside is security. While you are not sending all
your data to external devices, you are sending a model that is trained
with that data. While it is usually very hard to exploit, it is still a
security concern to consider.
Key Takeaways
1. Use batch inference where possible. Online inference should be a
last resort.
Conclusion
Investing in machine learning will enable you to solve business cases
that were previously impossible to solve, for example, automatically
categorizing images by their contents. Contrary to ML, MLOps doesn’t
come with a promise to solve any business problems directly. Instead,
it comes with the promise to accelerate how your investments in ML
return value.
For some, the MLOps practice is nothing new, especially for data
scientists with a strong software engineering background. In
collaborative efforts, however, it’s often essential to emphasize that
we work by best practices.
There are many different choices for MLOps tooling out on the
market and many different strategies to setting up your whole MLOps
toolchain, from building your own to buying a managed end-to-end
platform. In this eBook, we’ve tried to highlight the areas you should
consider and give some key takeaways when deciding what you need.
Conclusion 51
PREVIEW EDITION
Foreword Table of Contents
The MLOps scene is seeing tremendous growth from Automating custom model training
2 TBD 12
for document processing
all sides. Data scientists and machine learning
engineers are looking for better ways to work. Looking
at Google search volume for MLOps shows a growth Scoring backend for the NeurIPS
3 TBD 13
black-box optimization competition
trend that really kicked off in late 2019 and continues
to accelerate. In November 2020, we released an
Democratizing resources within a
eBook on MLOps and so far we’ve had over 2300 4 TBD 14
large data science team
individuals download it. This enthusiasm exceeded our
expectations and highlights for us the demand for
TBD 5 TBD 15
machine learning operations.
Team Size 3 people In Levity, users are using a series of pre-built The central challenge Levity faced was how to Part of the solution was designing the Levity
Time Spent 4 months integrations to connect their labeled data, train a build a system that delivers an excellent user product to require user input in the right places
new classifier to automate their manual PDF, experience that requires no knowledge of and in the right way. The user can adjust the
Models Trained 1,000
image, or text classification workflows. machine learning or associated terminology. The accepted error rate to clarify that there may be
Predictions Made 1,000,000
data upload needed to be dead simple so the outliers that are not represented in the training
Technologies Python After the user connects the training data, they user could populate a training data set for the data.
Tensorflow
specify how they want to receive the output of custom classifier.
Valohai
their custom classifier, for example directly into a The on-demand model training is at the core of
GCP
Google Sheet. The other part of an excellent UX was that the the product, and therefore the MLOps platform
trained, custom model would need to be usable needed to be deeply integrated. The Valohai API
The workflow is then connected to an input for quickly. The whole pipeline from data processing enabled just that allowing Levity to train and
Contact Person
new, unclassified data. From setup to usage, to training and deployment needed to happen deploy models via an API call and even rollback
Levity can train and deploy a custom classifier within a few minutes or less. And of course, this previous versions of the model if needed as all
Thilo Huellmann
within minutes for real world use. has to work consistently for every user. the artifacts produced are stored and versioned.
CTO & Co-Founder
at Levity
Black Box Black Box Black Box Black Box Black Box
User submits new Black Box Black Box Black Box Black Box Black Box
optimizer code on
the site. Black Box Black Box Black Box Black Box Black Box
Black Box Black Box Black Box Black Box Black Box
COMPETITION, WORLDWIDE
Black Box Black Box Black Box Black Box Black Box
Scoring backend for the
NeurIPS black-box opti-
mization competition
Team Size 4 people In the NeurIPS 2020 black-box optimization The challenge for the backend was simply the Valohai provided a robust backend where
Time Spent 3.5 months competition, the participants submit an massive amount of computing resources resources could automatically scale alongside
optimizer that needed to be scored immediately. required to score an optimizer. For optimal user the master scoring queue load. More than 300
Models Trained 277,000,000
To accurately score a black-box optimizer, the experience, the scoring needed to happen fast worker machines were crunching the numbers
Technologies Python
backend uses the optimizer to train several ML enough and with real-time intermediate scores. during peak hours – and at quieter moments, not
Bayesmark
models for thousands of iterations. a single one.
Valohai
The resources were absolutely required to be
The participants get intermediate scores while on-demand. The scoring queue load was As a bonus, the Valohai platform provided
the optimizer is being scored to improve the unpredictable, and static idle worker machines intuitive access to inspect the scoring tasks. This
stickiness of the competition. Waiting for a score would have been way too costly. helped a lot in a competition where the partici-
for too long would encourage participants to pants were not allowed to see the errors.
Contact Person
leave the competition prematurely. Organizers could still easily comment on
problems through manual channels as they had
Ryan Turner
access to full history.
Senior Research
Scientist at Twitter
Preligens can utilize
EARTH OBSERVATION, FRANCE
cloud and on-premise
Democratizing resources machines flexibly
depending on which is
within a large data the most economical.
Team Size 30 people Preligens has a large, growing data science team Preligens needed a flexible infrastructure layer to Preligens adopted Valohai to bring together the
Daily Experiments >10 experiments that works on various models embedded in their facilitate a large team doing daily deep learning whole team under a single tool. Three main
geospatial intelligence product. experiments involving satellite imagery. These features helped solve the team’s challenges.
Avg. Experiment 2 weeks
Runtime experiments may take days to complete, and
When a new data scientist starts at Preligens, therefore data scientists have to use remote Shared experiment tracking allowed data
Technologies Valohai
they want to be effective as soon as possible. hardware, i.e. cloud or on-premise machines scientists to collaborate. Each experiment is
The data scientist is onboarded to the shared with GPUs suited for the job. For this reason, automatically stored and reproducible by
MLOps platform on day 1 to ensure that s(he) has Preligens needed a multi-cloud approach. anyone.
access to past experiments.
Preligens wanted to integrate the MLOps With the platform handling the infrastructure
The newly onboarded data scientist usually platform with their proprietary framework, layer, data scientists could easily access remote
Contact Person
starts with running their first experiment in a which is used to facilitate work and centralize resources without DevOps knowledge.
sandbox project on the platform. The simple data, parameters, and other resources. For this
Renaud Allioux
example teaches how to utilize cloud machines reason, the infrastructure layer needed to be And finally, organization management allowed
Co-Founder & CTO
at Preligens without any help from IT or DevOps. fully controllable through an API. leadership to measure and manage costs.