The AI Product Manager's Handbook: Develop a product that takes advantage of machine learning to solve AI problems

BIRMINGHAM—MUMBAI

The AI Product Manager’s Handbook

Copyright © 2023 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author(s), nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Publishing Product Manager: Dinesh Chaudhary

Senior Editor: Tazeen Shaikh

Technical Editor: Rahul Limbachiya

Copy Editor: Safis Editing

Project Coordinator: Farheen Fathima

Proofreader: Safis Editing

Indexer: Pratik Shirodkar

Production Designer: Alishon Mendonca

Marketing Coordinators: Shifa Ansari and Vinishka Kalra

First published: February 2023

Production reference: 2230223

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham

B3 2PB, UK.

ISBN 978-1-80461-293-4

www.packtpub.com

For those courageous enough to believe they deserve their heart’s desires… evermore.

– Irene Bratsis

Contributors

About the author

Irene Bratsis is a director of digital product and data at the International WELL Building Institute (IWBI). She has a bachelor's in economics, and after completing various MOOCs in data science and big data analytics, she completed a data science program with Thinkful. Before joining IWBI, Irene worked as an operations analyst at Tesla, a data scientist at Gesture, a data product manager at Beekin, and head of product at Tenacity. Irene volunteers as NYC chapter co-lead for Women in Data, has coordinated various AI accelerators, moderated countless events with a speaker series with Women in AI called WaiTalk, and runs a monthly book club focused on data and AI books.

About the reviewer

Akshat Gurnani is a highly qualified individual with a background in the field of computer science and machine learning. He has a master’s degree in computer science and a deep understanding of various machine learning techniques and algorithms. He has experience working on various projects related to natural language processing, computer vision, and deep learning. He has also published several research papers in top-tier journals and conferences and has a proven track record in the field. He has a passion for keeping up to date with the latest developments in their fields and has a strong desire to continue learning and contributing to the field of artificial intelligence.

Table of Contents

Preface

Part 1 – Lay of the Land – Terms, Infrastructure, Types of AI, and Products Done Well

1

Understanding the Infrastructure and Tools for Building AI Products

Definitions – what is and is not AI

ML versus DL – understanding the difference

ML

DL

Learning types in ML

Supervised learning

Unsupervised learning

Semi-supervised learning

Reinforcement learning

The order – what is the optimal flow and where does every part of the process live?

Step 1 – Data availability and centralization

Step 2 – Continuous maintenance

Database

Data warehouse

Data lake (and lakehouse)

Data pipelines

Managing projects – IaaS

Deployment strategies – what do we do with these outputs?

Shadow deployment strategy

A/B testing model deployment strategy

Canary deployment strategy

Succeeding in AI – how well-managed AI companies do infrastructure right

The promise of AI – where is AI taking us?

Summary

Additional resources

References

2

Model Development and Maintenance for AI Products

Understanding the stages of NPD

Step 1 – Discovery

Step 2 – Define

Step 3 – Design

Step 4 – Implementation

Step 5 – Marketing

Step 6 – Training

Step 7 – Launch

Model types – from linear regression to neural networks

Training – when is a model ready for market?

Deployment – what happens after the workstation?

Testing and troubleshooting

Refreshing – the ethics of how often we update our models

Summary

Additional resources

References

3

Machine Learning and Deep Learning Deep Dive

The old – exploring ML

The new – exploring DL

Invisible influences

A brief history of DL

Types of neural networks

Emerging technologies – ancillary and related tech

Explainability – optimizing for ethics, caveats, and responsibility

Accuracy – optimizing for success

Summary

References

4

Commercializing AI Products

The professionals – examples of B2B products done right

The artists – examples of B2C products done right

The pioneers – examples of blue ocean products

The rebels – examples of red ocean products

The GOAT – examples of differentiated disruptive and dominant strategy products

The dominant strategy

The disruptive strategy

The differentiated strategy

Summary

References

5

AI Transformation and Its Impact on Product Management

Money and value – how AI could revolutionize our economic systems

Goods and services – growth in commercial MVPs

Government and autonomy – how AI will shape our borders and freedom

Sickness and health – the benefits of AI and nanotech across healthcare

Basic needs – AI for Good

Summary

Additional resources

References

Part 2 – Building an AI-Native Product

6

Understanding the AI-Native Product

Stages of AI product development

Phase 1 – Ideation

Phase 2 – Data management

Phase 3 – Research and development

Phase 4 – Deployment

AI/ML product dream team

AI PM

AI/ML/data strategists

Data engineer

Data analyst

Data scientist

ML engineer

Frontend/backend/full stack engineers

UX designers/researchers

Customer success

Marketing/sales/go-to-market team

Investing in your tech stack

Productizing AI-powered outputs – how AI product management is different

AI customization

Selling AI – product management as a higher octave of sales

Summary

References

7

Productizing the ML Service

Understanding the differences between AI and traditional software products

How are they similar?

How are they different?

B2B versus B2C – productizing business models

Domain knowledge – understanding the needs of your market

Experimentation – discover the needs of your collective

Consistency and AIOps/MLOps – reliance and trust

Performance evaluation – testing, retraining, and hyperparameter tuning

Feedback loop – relationship building

Summary

References

8

Customization for Verticals, Customers, and Peer Groups

Domains – orienting AI toward specific areas

Understanding your market

Understanding how your product design will serve your market

Building your AI product strategy

Verticals – examination into four areas (FinTech, healthcare, consumer goods, and cybersecurity)

FinTech

Healthcare

Cybersecurity

Anomaly detection and user and entity behavior analytics

Value metrics – evaluating performance across verticals and peer groups

Objectives and key results

Key performance indicators

Thought leadership – learning from peer groups

Summary

References

9

Macro and Micro AI for Your Product

Macro AI – Foundations and umbrellas

ML

Robotics

Expert systems

Fuzzy logic/fuzzy matching

Micro AI – Feature level

ML (traditional/DL/computer vision/NLP)

Robotics

Expert systems

Fuzzy logic/fuzzy matching

Successes – Examples that inspire

Lensa

PeriGen

Challenges – Common pitfalls

Ethics

Performance

Safety

Summary

References

10

Benchmarking Performance, Growth Hacking, and Cost

Value metrics – a guide to north star metrics, KPIs and OKRs

North star metrics

KPIs and other metrics

OKRs and product strategy

Hacking – product-led growth

The tech stack – early signals

Customer Data Platforms (CDPs)

Customer Engagement Platforms (CEPs)

Product analytics tools

A/B testing tools

Data warehouses

Business Intelligence (BI) tools

Growth-hacking tools

Managing costs and pricing – AI is expensive

Summary

References

Part 3 – Integrating AI into Existing Non-AI Products

11

The Rising Tide of AI

Evolve or die – when change is the only constant

The fourth industrial revolution – hospitals used to use candles

Working with a consultant

Working with a third party

The first hire

The first AI team

No-code tools

Fear is not the answer – there is more to gain than lose (or spend)

Anticipating potential risks

Summary

12

Trends and Insights across Industry

Highest growth areas – Forrester, Gartner, and McKinsey research

Embedded AI – applied and integrated use cases

Ethical AI – responsibility and privacy

Creative AI – generative and immersive applications

Autonomous AI development – TuringBots

Trends in AI adoption – let the data speak for itself

General trends

Embedded AI – applied and integrated use cases

Ethical AI – responsibility and privacy

Creative AI – generative and immersive applications

Autonomous AI development – TuringBots

Low-hanging fruit – quickest wins for AI enablement

Summary

References

13

Evolving Products into AI Products

Venn diagram – what’s possible and what’s probable

List 1 – value

List 2 – scope

List 3 – reach

Data is king – the bloodstream of the company

Preparation and research

Quality partnership

Benchmarking

The data team

Defining success

Competition – love your enemies

Product strategy – building a blueprint that works for everyone

Product strategy

Red flags and green flags – what to look for and watch out for

Red flags

Green flags

Summary

Additional resources

Index

Other Books You May Enjoy

Preface

It’s hard to come across anyone that doesn’t have strong opinions and reactions about AI these days. I’ve witnessed my own feelings and conclusions about it ebb and flow as the years have gone on. When I was a student, I felt a tremendous amount of excitement and optimism about where AI, and the fourth industrial revolution that accompanies it, would take us. That was quickly tempered when I started my book club, and I started a monthly practice of reading books about how bias and dependence on AI were compromising our lives in seen and unseen ways. Then, I started moderating events, where I brought together people from virtually every corner of AI and machine learning, who spoke not just on how they’re leveraging this technology in their own work but on their own beliefs about how AI will impact us in the future.

This brings us to one of the greatest debates we find ourselves returning to with every major advancement in technology. Do we dare adopt powerful technology even when we’re aware of the risks? As far as I see it, we don’t have a choice, and the debate is only an illusion we indulge ourselves in. AI is here to stay, and nihilistic fears about it won’t save us from any harm it may cause. Pandora’s box is open, and as we peer into what remains of it, we find that hope springs eternal.

AI is holding up a mirror to our biases and inequalities, and so far, it’s not a flattering reflection. It’s my hope that, with time, we will learn how to adopt AI responsibly in order to minimize its harm and optimize its greatest contributions to our modern civilization. I wanted to write a book about AI product management because it’s the makers of products that bring nebulous ideas into the real world. Getting into the details about how to ideate, build, manage and maintain AI products with integrity, to the best of my ability, is the greatest contribution I can make to this field at this present moment. It’s been an honor to write this book.

Who this book is for

This book is for people that aspire to be AI product managers, AI technologists, and entrepreneurs, or for people that are casually interested in the considerations of bringing AI products to life. It should serve you if you’re already working in product management and you have a curiosity about building AI products. It should also serve you if you already work in AI development in some capacity and you’re looking to bring those concepts into the discipline of product management and adopt a more business-oriented role. While some chapters in the book are more technically focused, all of the technical content in the book can be considered beginner level and accessible to all.

What this book covers

Chapter 1, Understanding the Infrastructure and Tools for Building AI Products, offers an overview of the main concepts and areas of infrastructure for managing AI products.

Chapter 2, Model Development and Maintenance for AI Products, delves into the nuances of model development and maintenance.

Chapter 3, Machine Learning and Deep Learning Deep Dive, is a broader discussion of the difference between traditional deep learning and deep learning algorithms and their use cases.

Chapter 4, Commercializing AI Products, discusses the major areas of AI products we see in the market, as well as examples of the ethics and success factors that contribute to commercialization.

Chapter 5, AI Transformation and Its Impact on Product Management, explores the ways AI can be incorporated into the major market sectors in the future.

Chapter 6, Understanding the AI-Native Product, gives an overview of the strategies, processes, and team building needed to empower the success of an AI-native product.

Chapter 7, Productizing the ML Service, is an exploration of the trials and tribulations that may come up when building an AI product from scratch.

Chapter 8, Customization for Verticals, Customers, and Peer Groups, is a discussion on how AI products change and evolve over various types of verticals, customer types, and peer groups.

Chapter 9, Macro and Micro AI for Your Product, gives an overview of the various ways you can leverage AI in ways big and small, as well as some of the most successful examples and common mistakes.

Chapter 10, Benchmarking Performance, Growth Hacking, and Cost, explains the benchmarking needed to gauge product success at the product level rather than the model performance level.

Chapter 11, The Rising Tide of AI, is a revisit to the concept of the fourth industrial revolution and a blueprint for products that don’t currently leverage AI.

Chapter 12, Trends and Insights across Industry, dives into the various ways we’re seeing AI trending across industries, based on prominent and respected research organizations.

Chapter 13, Evolving Products into AI Products, is a practical guide on how to deliver AI features and upgrade the existing logic of products to successfully update products for AI commercial success.

Conventions used

The text conventions used throughout this book are as follows:

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read The AI Product Manager’s Handbook, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/9781804612934

Submit your proof of purchase

That’s it! We’ll send your free PDF and other benefits to your email directly

Part 1 – Lay of the Land – Terms, Infrastructure, Types of AI, and Products Done Well

An AI product manager needs to have a comprehensive understanding of AI, along with all the varied components that lead to its success, if they’re going to be successful in commercializing their products.

This first part consists of five cumulative chapters that will cover what the term AI encompasses and how to support infrastructure to make it successful within your organization. It will also cover how to support your AI program from a maintenance perspective, how to navigate the vast areas of machine learning (ML) and deep learning (DL) and choose the best path for your product, and how to understand current and future developments in AI products.

By the end of this part, you will understand AI terms and components, what an AI implementation means from an investment perspective, how to maintain AI products sustainably, and how to choose between the types of AI that would best fit your product and market. You will also learn how to understand success factors for ideating and building a minimal viable product (MVP), and how to make a product that truly serves its market.

This part comprises the following chapters:

Chapter 1, Understanding the Infrastructure and Tools for Building AI Products

Chapter 2, Model Development and Maintenance for AI Products

Chapter 3, Machine Learning and Deep Learning Deep Dive

Chapter 4, Commercializing AI Products

Chapter 5, AI Transformation and Its Impact on Product Management

1

Understanding the Infrastructure and Tools for Building AI Products

Laying a solid foundation is an essential part of understanding anything, and the frontier of artificial intelligence (AI) products seems a lot like our universe: ever-expanding. That rate of expansion is increasing with every passing year as we go deeper into a new way to conceptualize products, organizations, and the industries we’re all a part of. Virtually every aspect of our lives will be impacted in some way by AI and we hope those reading will come out of this experience more confident about what AI adoption will look like for the products they support or hope to build someday.

Part 1 of this book will serve as an overview of the lay of the land. We will cover terms, infrastructure, types of AI algorithms, and products done well, and by the end of this section, you will understand the various considerations when attempting to build an AI strategy, whether you’re looking to create a native-AI product or add AI features to an existing product.

Managing AI products is a highly iterative process, and the work of a product manager is to help your organization discover what the best combination of infrastructure, training, and deployment workflow is to maximize success in your target market. The performance and success of AI products lie in understanding the infrastructure needed for managing AI pipelines, the outputs of which will then be integrated into a product. In this chapter, we will cover everything from databases to workbenches to deployment strategies to tools you can use to manage your AI projects, as well as how to gauge your product’s efficacy.

This chapter will serve as a high-level overview of the subsequent chapters in Part 1 but it will foremost allow for a definition of terms, which are quite hard to come by in today’s marketing-heavy AI competitive landscape. These days, it feels like every product is an AI product, and marketing departments are trigger-happy with sprinkling that term around, rendering it almost useless as a descriptor. We suspect this won’t be changing anytime soon, but the more fluency consumers and customers alike have with the capabilities and specifics of AI, machine learning (ML), and data science, the more we should see clarity about how products are built and optimized. Understanding the context of AI is important for anyone considering building or supporting an AI product.

In this chapter, we will cover the following topics:

Definitions – what is and is not AI

ML versus DL – understanding the difference

Learning types in ML

The order – what is the optimal flow and where does every part of the process live?

DB 101 – databases, warehouses, data lakes, and lakehouses

Managing projects – IaaS

Deployment strategies – what do we do with these outputs?

Succeeding in AI – how well-managed AI companies do infrastructure right

The promise of AI – where is AI taking us?

Definitions – what is and is not AI

In 1950, a mathematician and world war II war hero Alan Turing asked a simple question in his paper Computing Machinery and Intelligence – Can machines think?. Today, we’re still grappling with that same question. Depending on who you ask, AI can be many things. Many maps exist out there on the internet, from expert systems used in healthcare and finance to facial recognition to natural language processing to regression models. As we continue with this chapter, we will cover many of the facets of AI that apply to products emerging in the market.

For the purposes of applied AI in products across industries, in this book, we will focus primarily on ML and deep learning (DL) models used in various capacities because these are often used in production anywhere AI is referenced in any marketing capacity. We will use AI/ML as a blanket term covering a span of ML applications and we will cover the major areas most people would consider ML, such as DL, computer vision, natural language processing, and facial recognition. These are the methods of applied AI that most people will come across in the industry, and familiarity with these applications will serve any product manager looking to break into AI. If anything, we’d like to help anyone who’s looking to expand into the field from another product management background to choose which area of AI appeals to them most.

We’d also like to cover what is and what isn’t ML. The best way for us to express it as simply as we can is: if a machine is learning from some past behavior and if its success rate is improving as a result of this learning, it is ML! Learning is the active element. No models are perfect but we do learn a lot from employing models. Every model will have some element of hyperparameter tuning, and the use of each model will yield certain results in performance. Data scientists and ML engineers working with these models will be able to benchmark performance and see how performance is improving. If there are fixed, hardcoded rules that don’t change, it’s not ML.

AI is a subset of computer science, and all programmers are effectively doing just that: giving computers a set of instructions to fire away on. If your current program doesn’t learn from the past in any way, if it simply executes on directives it was hardcoded with, we can’t call this ML. You may have heard the terms rules-based engine or expert system thrown around in other programs. They are considered forms of AI, but they're not ML because although they are a form of AI, the rules are effectively replicating the work of a person, and the system itself is not learning or changing on its own.

We find ourselves in a tricky time in AI adoption where it can be very difficult to find information online about what makes a product AI. Marketing is eager to add the AI label to their products but there still isn’t a baseline of explainability with what that means out in the market. This further confuses the term AI for consumers and technologists alike. If you’re confused by the terms, particularly when they’re applied to products you see promoted online, you’re very much not alone.

Another area of confusion is the general term that is AI. For most people, the concept of AI brings to mind the Terminator franchise from the 1980s and other futurist depictions of inescapable technological destruction. While there certainly can be a lot of harm to come from AI, this depiction represents what’s referred to as strong AI or artificial general intelligence (AGI). We still have ways to go for something such as AGI but we’ve got plenty of what’s referred to as artificial narrow intelligence or narrow AI (ANI).

ANI is also commonly expressed as weak AI and is what’s generally meant when you see AI plastered all over products you find online. ANI is exactly what it sounds like: a narrow application of AI. Maybe it’s good at talking to you, at predicting some future value, or at organizing things; maybe it’s an expert at that, but its expertise won’t bleed into other areas. If it could, it would stop being ANI. These major areas of AI are referred to as strong and weak in comparison to human intelligence. Even the most convincing conversational AIs out there, and they are quite convincing, are demonstrating an illusionary intelligence. Effectively, all AI that exists at the moment is weak or ANI. Our Terminator days are still firmly in our future, perhaps never to be realized.

For every person out there that’s come across Reddit threads about AI being sentient or somehow having ill will toward us, we want to make the following statement very clear. AGI does not exist and there is no such thing as sentient AI. This does not mean AI doesn’t actively and routinely cause humans harm, even in its current form. The major caveat here is that unethical, haphazard applications of AI already actively cause us both minor inconveniences and major upsets. Building AI ethically and responsibly is still a work in progress. While AI systems may not be sentiently plotting the downfall of humanity, when they’re left untested, improperly managed, and inadequately vetted for bias, the applications of ANI that are deployed already have the capacity to do real damage in our lives.

For now, can machines think like us? No, they don’t think like us. Will they someday? We hope not. It’s my personal opinion that the insufferable aspects of the human condition end with us. But we do very much believe that we will experience some of our greatest ails, as well as our wildest curiosities, to be impacted considerably by the benevolence of AI and ML.

ML versus DL – understanding the difference

As a product manager, you’re going to need to build a lot of trust with your technical counterparts so that, together, you can build an amazing product that works as well as it can technically. If you’re reading this book, you’ve likely come across the phrase ML and DL. We will use the following sections titled ML and DL to go over some of the basics but keep in mind that we will be elaborating on these concepts further down in Chapter 3.

ML

In its basic form, ML is made up of two essential components: the models used and the training data it’s learning from. These data are historical data points that effectively teach machines a baseline foundation from which to learn, and every time you retrain the models, the models are theoretically improving. How the models are chosen, built, tuned, and maintained for optimized performance is the work of data scientists and ML engineers. Using this knowledge of performance toward the optimization of the product experience itself is the work of product managers. If you’re working in the field of AI product management, you’re working incredibly closely with your data science and ML teams.

We’d like to also make a distinction about the folks you’ll be working with as an AI product manager. Depending on your organization, you’re either working with data scientists and developers to deploy ML or you’re working with ML engineers who can both train and upkeep the models as well as deploy them into production. We highly suggest maintaining strong relationships with any and all of these impacted teams, along with DevOps.

All ML models can be grouped into the following four major learning categories:

Supervised learning

Unsupervised learning

Semi-supervised learning

Reinforcement learning

These are the four major areas of ML and each area is going to have its particular models and algorithms that are used in each specialization. The learning type has to do with whether or not you’re labeling the data and the method you’re using to reward the models you’ve used for good performance. These learning types are relevant whether your product is using a DL model or not, so they’re inclusive of all ML models. We will be covering the learning types in more depth in the following section titled Learning types in ML.

DL

DL is a subset of ML, but the terms are often used colloquially as almost separate expressions. The reason for this is DL is based on neural network algorithms and ML can be thought of as… the rest of the algorithms. In the preceding section covering ML, we looked at the process of taking data, using it to train our models, and using that trained model to predict new future data points. Every time you use the model, you see how off it was from the correct answer by getting some understanding of the rate of error so you can iterate back and forth until you have a model that works well enough. Every time, you are creating a model based on data that has certain patterns or features.

This process is the same in DL, but one of the key differences of DL is that patterns or features in your data are largely picked up by the DL algorithm through what’s referred to as feature learning or feature engineering through a hierarchical layered system. We will go into the various algorithms that are used in the following section because there are a few nuances between each, but as you continue developing your understanding of the types of ML out there, you’ll also start to group the various models that make up these major areas of AI (ML and DL). For marketing purposes, you will for the most part see terms such as ML, DL/neural networks, or just the general umbrella term of AI referenced where DL algorithms are used.

It’s important to know the difference between what these terms mean in practice and at the model level and how they’re communicated by non-technical stakeholders. As product managers, we are toeing the line between the two worlds: what engineering is building and what marketing is communicating. Anytime you’ve heard the term black box model, it’s referring to