Professional Documents
Culture Documents
Principles of Web API Design - Delivering Value With APIs and Microservices
Principles of Web API Design - Delivering Value With APIs and Microservices
“I’ve had the good fortune to work alongside and learn from James over the past several years.
His varied institutional knowledge, along with his depth of experience and eye for practical
application, makes him unique among his peers. I am ecstatic that others now have the opportu-
nity, in this book, to benefit from James’s compelling, pragmatic vision for how to make better
APIs. Principles of Web API Design surveys the gamut of available techniques and sets forth a
prescriptive, easy-to-follow approach. Teams that apply the guidance in this book will create
APIs that better resonate with customers, deliver more business value in less time, and require
fewer breaking changes. I cannot recommend Principles of Web API Design enough.”
“James is one of the preeminent experts on API design in the industry, and this comprehensive
guide reflects that. Putting API design in the context of business outcomes and digital capabili-
ties makes this a vital guide for any organization undergoing digital transformation.”
“In modern software development, APIs end up being both the cause of and solution to many of
the problems we face. James’s process for dissecting, analyzing, and designing APIs from concepts
to caching creates a repeatable approach for teams to solve more problems than they create.”
“Following James’s clear and easy-to-follow guide, in one afternoon I was able to apply his
process to current real-world use cases. I now have the practical guidance, techniques, and
clear examples to help me take those next vital steps. Recommended reading for anyone con-
nected to and working with APIs.”
“Principles of Web API Design uncovers more than principles. In it, you’ll learn a process—a
method to design APIs.”
“This insightful playbook guides API teams through a structured process that fosters produc-
tive collaboration, valuable capability identification, and best-practice contract crafting.
James distills years of experience into a pragmatic roadmap for defining and refining API
products, and also provides a primer for API security, eventing, resiliency, and microservices
alignment. A must-read for architects either new to the API discipline or responsible for
onboarding new teams and instituting a structured API definition process.”
James Higginbotham
Figure 10.12, icons: dependency by Knut M. Synstad from the Noun Project; plug by Vectors Market from the Noun Project;
database by MRK from the Noun Project; filter by Landan Lloyd from the Noun Project; command line by Focus from the
Noun Project; algorithm by Trevor Dsouza from the Noun Project; name tag by Cindy Clegane from the Noun Project; task
list by Royal@design from the Noun Project; quality by Flatart from the Noun Project; broadcast by Yoyon Pujiyono from the
Noun Project.
Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where
those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed
with initial capital letters or in all capitals.
The author and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any
kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in
connection with or arising out of the use of the information or programs contained herein.
For information about buying this title in bulk quantities, or for special sales opportunities (which may include electronic
versions; custom cover designs; and content particular to your business, training goals, marketing focus, or branding
interests), please contact our corporate sales department at [email protected] or (800) 382-3419.
For questions about sales outside the U.S., please contact [email protected].
All rights reserved. This publication is protected by copyright, and permission must be obtained from the publisher prior
to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic,
mechanical, photocopying, recording, or likewise. For information regarding permissions, request forms and the appropriate
contacts within the Pearson Education Global Rights & Permissions Department, please visit www.pearson.com/permissions.
ISBN-13: 978-0-13-735563-1
ISBN-10: 0-13-735563-7
ScoutAutomatedPrintCode
Pearson’s Commitment to Diversity, Equity, and Inclusion
Pearson is dedicated to creating bias-free content that reflects the diversity of all
learners. We embrace the many dimensions of diversity, including but not limited to
race, ethnicity, gender, socioeconomic status, ability, age, sexual orientation, and
religious or political beliefs.
Education is a powerful force for equity and change in our world. It has the
potential to deliver opportunities that improve lives and enable economic mobility.
As we work with authors to create content for every product and service, we
acknowledge our responsibility to demonstrate inclusivity and incorporate diverse
scholarship so that everyone can achieve their potential through learning. As the
world’s leading learning company, we have a duty to help drive change and live up
to our purpose to help more people create a better life for themselves and to create
a better world.
Our ambition is to purposefully contribute to a world where:
While we work hard to present unbiased content, we want to hear from you about
any concerns or needs with this Pearson product so that we can investigate and
address them.
To my grandfather, J.W.,
who gave me a Commodore 64 when I was eight years old
because he believed “computers are going to be big someday,
and my grandson should know how to use one”
and who inspired me to follow in his footsteps as an author.
To my dad,
who continued the work
of J.W. I miss you.
To my son,
who continues the tradition with
his endless coding in Minecraft.
And to my daughter,
who inspires me every day
to write better copy.
This page intentionally left blank
Contents
xi
xii Contents
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319
This page intentionally left blank
Series Editor Foreword
xxi
xxii Series Editor Foreword
—Vaughn Vernon
This page intentionally left blank
Foreword
According to a recent IDC report on APIs and API management, 75 percent of those
surveyed were focused on digital transformation through the design and implemen-
tation of APIs and more than one half expected call volume and response time to
grow dramatically. And most organizations admitted they faced challenges in meet-
ing expectations for both internally and externally facing APIs. At the heart of all of
this is the need for consistent, reliable, and scalable API design programs to help lead
and transform existing organizations. As James Higginbotham puts it in this book:
“The biggest challenge for today’s API programs continues to be successfully design-
ing APIs that can be understood and integrated by developers in a consistent and
scalable fashion.”
It was for this reason that I was so happy to have this book cross my desk. I’ve had
the pleasure of working with James over the years and, knowing his work and his
reputation, was very happy to hear he was writing a book that covers Web API
design. Now, after reading through this book, I am equally happy to recommend it to
you, the reader.
The field of Web APIs and the work of designing them has matured rapidly over
the last few years, and keeping up with the latest developments is a major undertak-
ing. Issues like changing business expectations for the role of APIs; maturing pro-
cesses for gathering, recording, and documenting the work of API design; as well as
evolving technology changes and all the work of coding, releasing, testing, and mon-
itoring APIs make up an API landscape large enough that few people have been able
to successfully tackle it. Through his Align-Define-Design-Refine process, James
offers an excellent set of recommendations, examples, and experience-based advice
to help the reader navigate the existing space of Web APIs and prepare for the inevi-
table changes ahead in the future.
One of the things about James’s work that has always stood out is his ability to
reach beyond the technical and into the social and business aspects of APIs and API
programs within organizations. James has a long list of international clients across
the business sectors of banking, insurance, global shipping, and even computer
hardware providers, and the material in this book reflects this depth of experience.
The techniques and processes detailed here have been tried and tested in all sorts of
enterprise settings, and James’s ability to distill what works into this one volume is
xxv
xxvi Foreword
impressive. Whether you are looking for advice on general design, business-
technology alignment, or implementation details for various technologies such as
REST, GraphQL, and event-driven platforms, you’ll find important and actionable
advice within these pages.
In particular, I found the material on how to refine your API design and imple-
mentation efforts within an ever-growing enterprise API program particularly timely
and especially valuable. For those tasked with launching, managing, and expanding
the role of Web-based APIs within a company, Principles of Web API Design should
prove to be a welcome addition to your bookshelf.
As the aforementioned IDC report indicates, many companies around the globe
are faced with important digital transformation challenges, and APIs have a major
role to play in helping organizations meet the needs of their customers and in con-
tinuing to improve their own bottom line. Whether you are focused on designing,
building, deploying, or maintaining APIs, this book contains helpful insights and
advice.
I know this book will become an important part of my toolkit as I work with
companies of all stripes to continue to mature and grow their API programs, and I
expect you, too, will find it useful. Reading this book has reminded me of all the
opportunities and challenges we all have before us. To borrow another line from
James: “This is only the beginning.”
It’s hard to pinpoint the beginning of the journey to writing this book—perhaps it
started about ten years ago. It is the result of thousands of hours of training, tens of
thousands of miles traveled, and too many written words and lines of code to count.
It comprises insights from organizations across the globe that were just starting their
API journey or had already begun the adventure. The book incorporates the insights
of API practitioners across the world whom I have had the pleasure to meet.
Or perhaps the journey started almost twenty-five years ago, when I first entered
the software profession. So many advisors provided their insight via books and arti-
cles. Mentors along the way helped to shape my way of thinking about software.
They laid the foundation of how I prefer to realize software architecture.
Maybe the journey really started almost forty years ago, when my grandfather
gifted me with a Commodore 64. He was a civil engineer and cost engineer who
attended night school while working to support his family during the day. He was
thirsty for knowledge, reading and absorbing everything he could. He always made
us laugh when he said, “I’m still amazed at how television works!” after seeing a
computer operate. Yet, he was the one who gifted me that magical computer, saying
“computers are going to be big someday, and my grandson should know how to use
one.” This single action started my lifelong love of software development.
In reality, the journey started more than seventy years ago when the pioneers of
our current age of computing established many of the foundational principles we
still use today to construct software. Though technology choices change, and the
trends come and go, it all builds on the work of so many in the software industry and
beyond. Countless people have helped to carve the way for what we do today.
What I am saying is that APIs would not be what they are today without all the
hard work that came before us. Therefore, we must thirst for understanding the
history of our industry to better understand “the how” and “the why” behind what
we do today. Then, we must seek to apply these lessons to all that we do tomorrow.
Along the way, we need to find ways to inspire others to do the same. This is what
my grandfather and father taught me, so I pass this lesson on to you. This book
reflects the things I’ve learned thus far in my journey. I hope you gain some new
insights by building upon what is presented here while you seek to prepare the next
generation.
xxvii
xxviii Preface
• Part V: Refining the Design—Improves the API design based on insights from
documentation, testing, and feedback. It also includes a chapter on decompos-
ing APIs into microservices. Finally, the book closes with tips on how to scale
the design process in larger organizations.
For those who need a refresher on HTTP, the language of the Web used for Web-
based APIs, the appendix provides a nice primer to help you get started.
Register your copy of Principles of Web API Design on the InformIT site for
convenient access to updates and/or corrections as they become available. To
start the registration process, go to informit.com/register and log in or create
an account. Enter the product ISBN (9780137355631) and click Submit. Look
on the Registered Products tab for an Access Bonus Content link next to this
product, and follow that link to access any available bonus materials. If you
would like to be notified of exclusive offers on new editions and updates,
please check the box to receive email from us.
This page intentionally left blank
Acknowledgments
First, I would like to thank my wife and kids who have supported me in so many ways
throughout the years. Your prayers and encouragement have meant so much to me.
Special thanks to Jeff Schneider, who suggested that we should write the first
enterprise Java book in 1996, before Java was enterprise. Your insights and endless
hours of coaching set me on an amazing career path. Your friendship guided me
along the way.
Keith Casey, thank you for inviting me to coauthor a book and deliver API
workshops to people all over the world. This book wouldn’t have been written
without your friendship, encouragement, and insight.
Vaughn Vernon, who sent me a message years ago asking how we could collabo-
rate, which ultimately turned into this book—thank you for inviting me on your
journey.
Mike Williams, who encouraged me to risk it all to realize my dreams, you have
been an inspiration and a great friend.
A special thank you to the many reviewers of this book. Your dedication to
reviewing the chapters, often under a time crunch, to help produce this book is
appreciated: Mike Amundsen, Brian Conway, Adam DuVander, Michael Hibay,
Arnaud Lauret, Emmanuel Paraskakis, Matthew Reinbold, Joyce Stack, Vaughn
Vernon, and Olaf Zimmermann.
To all API evangelists and influencers, thank you for the personal and professional
discussions. Here are just a few of the many people I’ve had the pleasure of meeting:
Tony Blank, Mark Boyd, Lorinda Brandon, Chris Busse, Bill Doerfeld, Marsh Gardiner,
Dave Goldberg, Jason Harmon, Kirsten Hunter, Kin Lane, Matt McLarty, Mehdi
Medjaoui, Fran Mendez, Ronnie Mitra, Darrel Miller, John Musser, Mandy Whaley,
Jeremy Whitlock, and Rob Zazueta. And to those on the Slack channel, thanks for your
support!
I would like to acknowledge everyone at Pearson who supported me throughout
the process. Haze Humbert, thank you for making this process as easy as it can be for
an author. And thank you to the entire production team: your hard work is greatly
appreciated.
Finally, to my mom, thank you for spending endless hours at the library while
I researched computer programming books before I was old enough to drive.
xxxi
This page intentionally left blank
About the Author
James Higginbotham is a software developer and architect with over twenty-five years
of experience in developing and deploying apps and APIs. He guides enterprises
through their digital transformation journey, ensuring alignment between business and
technology through product-based thinking to deliver a great customer experience.
James engages with teams and organizations to help them align their business, prod-
uct, and technology strategies into a more composable and modular enterprise plat-
form. James also delivers workshops that help cross-functional teams to apply an API
design-first approach using his ADDR process. His industry experience includes bank-
ing, commercial insurance, hospitality, travel, and the airline industry where he helped
to get an airline off the ground—literally. You can learn more about his latest efforts at
https://launchany.com and on Twitter @launchany.
xxxiii
This page intentionally left blank
Part I
Introduction to Web
API Design
APIs are forever. Once an API is integrated into a production application, it is
difficult to make significant changes that could potentially break those existing inte-
grations. Design decisions made in haste become future areas of confusion, support
issues, and lost opportunities far into the future. The API design phase is an impor-
tant part of any delivery schedule.
Part 1 examines the fundamentals of software design and how it produces a posi-
tive or negative impact on API design. It then examines the API first design process
and presents an overview of an API design process. This process incorporates an
outside-in perspective to deliver an effective API to meet the needs of customers,
partners, and the workforce.
1
This page intentionally left blank
Chapter 1
The Principles
of API Design
Organizations have been delivering APIs for decades. APIs started as libraries and
components shared across an organization and sold by third parties. They then grew
into distributed components using standards such as CORBA for distributed object
integration and SOAP for integrating distributed services across organizations.
These standards were designed for interoperability but lacked the elements of effec-
tive design, often requiring months of effort to successfully integrate them.
As these standards were replaced by Web APIs, only a few APIs were needed.
Teams could take the time to properly design them, iterating as needed. This is no
longer the case. Organizations deliver more APIs and at greater velocity than ever
before. The reach of Web APIs goes beyond a few internal systems and partners.
Today’s Web-based APIs connect organizations to their customers, partners,
and workforce using the standards of the Web. Hundreds of libraries and frame-
works exist to make it cheap and fast to deliver APIs to a marketplace or for internal
use. Continuous integration and continuous delivery (CI/CD) tools make it easier
than ever to build automation pipelines to ensure APIs are delivered with speed and
efficiency.
Yet, the biggest challenge for today’s API programs continues to be successfully
designing APIs that can be understood and integrated by developers in a consistent
and scalable fashion. Facing this challenge requires organizations to recognize that
Web APIs are more than just technology. Just as works of art require the balance
of color and light, API design benefits from the blending of business capabilities,
product thinking, and a focus on developer experience.
3
4 Chapter 1 The Principles of API Design
Business Capabilities
Business capabilities describe the enablers an organization brings to market. They
may include external-facing capabilities, such as unique product design, amazing
customer service, or optimized product delivery. They may also include internally
facing capabilities such as sales pipeline management or credit risk assessment.
Organizations deliver business capabilities in three ways: directly by the organiza-
tion, outsourced via a third-party provider, or through a combination of organiza-
tional and third-party processes.
For example, a local coffee shop may choose to sell custom coffee blends. To
do so, it sources coffee beans through a third-party distributor, roasts the coffee
beans in-house, then utilizes a third-party point-of-sale (POS) system for selling
its coffee blends in a retail store. By outsourcing some of the necessary busi-
ness capabilities to specialized third parties, the coffee shop is able to focus on
delivering specific business capabilities that differentiate them from others in the
marketplace.
APIs digitize the business capabilities that an organization brings to a market-
place. When embarking on designing a new API or expanding an existing API, the
underlying business capabilities should be well understood and reflected into the
API design.
Product Thinking
Organizations were integrating with partners and customers prior to the growth of
Web APIs. The challenge most organizations face, however, is that each integration
has been custom made. For each new partner or customer integration, a dedicated
team consisting of developers, a project manager, and an account manager were
tasked with building a custom integration. This involved tremendous effort and was
often repeated, with per-partner customizations.
The Elements of Web API Design 5
The growth of the software-as-a-service (SaaS) business model, along with the
increase in demand for Web APIs, have shifted the discussion from one-off integra-
tion with partners and customers to a focus on product thinking.
Applying product thinking to the API design process shifts the team focus from
a single customer or partner to an effective API design that is able to handle new
automation opportunities with little to no customization effort for a given customer
segment. It also enables a self-service model for workforce, business-to-business, and
customer-driven integration.
The focus of an API product becomes less on custom implementations and more
on meeting market needs in a scalable and cost-effective way. Reusable APIs emerge
from considering multiple consumers at once. When embarking on the design of a
new API, use a product thinking approach to obtain feedback from multiple par-
ties that will consume the API. Doing so will shape the API design early and lead to
increased opportunities for reuse.
Developer Experience
User experience (UX) is the discipline of meeting the exact needs of users, from their
interactions with the company to their interactions with its services and with the
product itself. Developer experience (DX) is just as important for APIs as UX is for
products and services. The DX focuses on the various aspects of engagement with
developers for an API product. It extends beyond the operational details of the API.
It also includes all aspects of the API product, from first impressions to day-to-day
usage and support.
A great DX is essential to the success of an API. When a great DX is delivered,
developers quickly and confidently consume a Web API. It also improves the mar-
ket traction of productized APIs by moving developers from being integrators to
becoming experts on the API. The expertise translates directly into the ability to
deliver real value to their customers and their business quickly and with reduced
effort.
As API teams seek to understand how to design a great experience for their API,
remember that DX is an important factor for internal developers, also. For example,
great documentation enables internal developers to understand and consume an API
quickly, whereas an API that has poor documentation requires contacting the inter-
nal team responsible for the API to learn how to use it properly. While they may be
able to gain direct access to the developers that designed and implemented an API,
it adds unnecessary communication overhead. Internal developers benefit from great
DX because they can create business value faster.
6 Chapter 1 The Principles of API Design
CASE STUDY
APIs and Product Thinking Meets Banking
Capital One started its API journey in 2013 with the goal of developing an
enterprise API platform. The initial set of platform APIs focused on deliver-
ing automation throughout the organization to increase velocity of delivery
while breaking down siloed barriers.
As the number of digital capabilities in its API platform grew, Capital
One’s focus shifted from internal APIs to several product opportunities
in the marketplace. It launched its public-facing developer portal called
DevExchange at South by Southwest (SXSW)1 with several API products.
These product offerings included bank-grade authorization, a rewards
program, credit card prequalification, and even an API to create new sav-
ings accounts.
Capital One extended the idea further by leveraging its digital capabili-
ties to develop an omnichannel presence. APIs used to power its Web site and
mobile app formed a foundation for a voice-based interactive experience2
using Amazon’s Alexa platform and interactive chat using a chatbot named
Eno (the word one spelled backwards).
Taking a product-based approach to its APIs, along with a robust API
portfolio of digital capabilities, allowed Capital One to explore opportunities
with its customers and partners. It didn’t happen overnight, but it did happen
because of an API focus that started with an executive vision and execution
by the entire organization.
Modularization
Modules are the smallest atomic unit within a software program. They are com-
posed of one or more source files that contain classes, methods, or functions. Mod-
ules have a local, public API to expose the functionality and business capabilities that
they offer to other modules within the same codebase. Modules are sometimes
referred to as components or code libraries.
Most programming languages support modules through the use of namespaces
or packages that group code together. Grouping related code that collaborates into
the same namespace encourages high cohesion. Internal details of a module are
protected through access modifiers provided by the programming language. For
example, the Java programming language has keywords such as pu b lic, protected,
pack ag e, and priv ate that help to encourage loose coupling through limited expo-
sure of a module.
As more and more modules are combined, a software system is created. A sub-
system combines modules into a larger module in more complex solutions, as
shown in Figure 1.1.
Applying the same concepts of modularization to Web-based API design helps to
reveal the boundaries and responsibilities of every API. This ensures clear responsi-
bilities across complementary APIs that focus on externalizing digital capabilities
while hiding the internal implementation details. Consuming developers benefit by
understanding the API quickly and effectively.
Encapsulation
Encapsulation seeks to hide the internal details of a component. Scope modifiers are
used to limit access to a module’s code. A module exposes a set of public methods
or functions while hiding the internal details of the module. Internal changes may
System
Subsystem Subsystem
Module Module
Module Module
Figure 1.1 Modules combine into ever-larger units, resulting in a software system.
Reviewing the Principles of Software Design 9
occur without impacting other modules that depend on its public methods. Some-
times encapsulation is referred to as information hiding, a concept applied to soft-
ware development since the 1970s by David Parnas.
Web APIs extend this concept a bit further. They hide the internal details of pro-
gramming language, choice of Web framework, the classes and objects of a system,
and database design behind an HTTP-based API. Internal details, encapsulated
behind the API design, encourage a loosely coupled API design that depends on mes-
sages rather than underlying database design and models for communication. No
longer do organizations need to understand all the internal implementations details,
such as for a payment gateway. Instead, they only need to understand the operations
that the API offers and how to use them to achieve the desired outcomes.
...
Web APIs extend these concepts by grouping related API operations for high
cohesion while ensuring that the internal details are encapsulated to encourage a
loosely coupled API design.
I’m sorry that I long ago coined the term “objects” for this topic because it gets many
people to focus on the lesser idea.
The big idea is “messaging.”3
Like Kay’s original vision for object-oriented programming, Web APIs are mes-
sage based. They send request messages to a server and receive a response message as
a result. Most Web APIs perform this message exchange synchronously by sending a
request and waiting for the response.
API design considers the conversational message exchange between systems to
produce desired outcomes by customers, partners, and the workforce. A great API
design also considers how this communication evolves as requirements change.
3. Alan Kay, “Prototypes vs Classes was: Re: Sun’s HotSpot,” Squeak Developer’s List, October 10, 1998,
http://lists.squeakfoundation.org/pipermail/squeak-dev/1998-October/017019.html.
The Principles of Web API Design 13
Today 8:32 AM
Today 8:32 AM
Today 8:32 AM
Figure 1.3 An example interaction between an API client and API server, as if the user was
talking to the server in conversational terms.
Summary
Web API design incorporates three important elements to deliver a successful API:
business capabilities, product thinking, and developer experience. These cross-
functional disciplines mean that organizations cannot ignore the process of API
design. Developers, architects, domain experts, and product managers must work
together to design APIs that meet the needs of the marketplace.
In addition, Web API design builds on the principles of software design, includ-
ing modularization, encapsulation, loose coupling, and high cohesion. API designs
should hide the internal details of the systems they externalize. They should not
expose underlying data models but rather focus on a system-to-system message
exchange that is both flexible in design and resilient to change over time.
So, how do teams go from business requirements to an API design that is evolvable
while delivering the desired outcomes to customers, partners, and the internal work-
force? That is the subject of the next chapter, which introduces a process that bridges
business and product requirements into an API design. The process is explored in
detail in subsequent chapters.
Chapter 2
Big design up front is dumb but doing no design up front is even dumber.
— Dave Thomas
An API design that looks good to the designer may not be the best design to solve the
problems at hand. The initial assumptions about an API design may be incorrect as
the API encounters the real world of customer, partner, and workforce needs.
API contract design is a separate and critical step of software delivery. Following
an API design process encourages communication internally within the organization
and externally between the organization and the developers ultimately tasked
with integrating the API. It helps to identify incorrect assumptions and validate
the assumptions that are correct. Finally, it encourages collaboration between API
designers and the developers that will integrate the API.
This chapter presents a design process that is flexible to meet the needs of a single
API product or mid- to large-scale enterprise API platform. Organizations from as
small as 10 employees to those with a staff of more than 10,000 developers have
used this collaborative design process. Ultimately, it delivers business value with
a customer-centric focus by applying outside-in design using the five principles
outlined in Chapter 1, “The Principles of API Design.”
15
16 Chapter 2 Collaborative API Design
multiple iterations of breaking design changes. The APIs that weren’t properly
designed lacked sufficient insights into how to use the API compared to those
designed with an API design process.
An API design process encourages efficiency throughout the delivery process.
By focusing on the API contract first, the design represents the needs of users and
developers as a primary concern. Also, implementation details are less likely to leak
into the API design, resulting in a fragile API design that must introduce breaking
changes as implementation details change over time.
A backend API is the primary blocker for any frontend delivery schedule. If
frontend developers are forced to wait until the backend developers have completed
the API implementation, the end-to-end delivery process will take too long. Any
errors in design won’t be identified until the frontend developers start to integrate
the API. Customer feedback isn’t available until all of the integration work has
been completed. Figure 2.1 visualizes this problem and the impact it has on the
delivery schedule.
An API design process encourages an iterative, team-oriented design effort that
allows for greater overall efficiency. The frontend and backend API teams work
together to arrive at a design, then parallelize their specific tasks. Customer feedback
may be incorporated earlier as well, avoiding last-minute rework efforts. As depicted
in Figure 2.2, the process is repeated for each release, ensuring the design process
becomes more rapid while incorporating feedback iteratively. Remember that the
sooner that API design mistakes are caught, the cheaper they are to fix.
T1 T2 T3
Figure 2.1 The negative impact of delivering APIs in isolation. The minimum calendar time required is T1 + T2 + T3.
API Design Process Antipatterns
17
18
Chapter 2
Release 1 Release 2
T1 T2
T3
Figure 2.2 The positive impact of delivering APIs with a design process that optimizes for efficiency and repeatability. The minimum calendar
time required is T1 + max(T2 + T3).
API Design Process Antipatterns 19
While an API design process does not guarantee a perfect design the first time, it
helps challenge assumptions quickly. It also encourages early communication with
subject matter experts and customers to address flawed design issues before they are
too expensive to rectify.
1. Discover: Determine the digital capabilities the API needs to deliver, searching
for APIs that may already exist to meet the requirements.
2. Design: Produce an initial API design or improve an existing API design to
address the digital capabilities required but not available.
The API Design-First Approach 21
4. Deliver
5. Onboard
Notice the iterative design process that occurs as stakeholder input is gath-
ered. Feedback is incorporated early and often, making design changes along the
way. This results in an API contract, which provides the specific details of how the
design is realized. Prototypes or mock implementations demonstrate the API in
action, prior to the full delivery process. Once delivery begins, the effort is paral-
lelized across all teams, with the API contract as the primary communication arti-
fact. After developers are onboarded with the API, additional feedback results in a
new design effort.
Keeping these principles in mind, teams have the opportunity to remain agile
while communicating early and often with stakeholders on the API design. These
stakeholders may include internal development teams, channel partners, and the
developers tasked with integrating the API.
Delivering an API design progressively, rather than all at once, allows teams to
meet the principle of welcoming changing requirements and delivering working
software frequently. It also helps teams avoid last-minute scrambles that can
negatively impact API design.
The “seek the simple” principle encourages teams to design in a simple way.
Teams should design an API that avoids clever designs that require a higher cognitive
load to understanding. Instead, designs should be intuitive based on the use cases
it addresses and should use vocabulary that is appropriate for the solution domain.
They should offer only the necessary information to support the use case.
Remember
Teams can always add to an API design, but it is impossible to take things away
without breaking integrations that depend on them. Take advantage of agile
software development to incrementally design APIs with the needs of customers,
partners, and the workforce in mind.
2. The ADDR process is based on the many lessons learned during my years of experience in API design
coaching.
24 Chapter 2 Collaborative API Design
3. Design: Applies specific design steps for each API to meet the desired outcomes
using one or more API styles
4. Refine: Refines the API design through feedback from developers, in addition
to documentation, prototyping, and testing efforts
There are seven steps across the phases, which are explored in-depth for the
remainder of this book:
1. Identify digital capabilities: Identify the customer needs and desired out-
comes, including the corresponding digital capabilities that are required.
2. Capture activity steps: Expand the digital capabilities to include a unified
understanding and clarity through collaborative API design sessions.
3. Identify API boundaries: Group the digital capabilities into API boundaries
and determine whether the APIs already exist or new APIs are required.
4. Model API profiles: Use a collaborative API modeling session to define the
high-level API design, including resources and operations into an API profile.
5. High-level API designs: Select one or more API styles that each API profile
will offer and document the high-level design elements.
6. Refine the design: Incorporate design feedback from API consumers using
techniques that encourage improvement in the developer experience.
7. Document the API: Complete the API documentation, including reference
documentation and getting started guides, to accelerate integration.
Figure 2.4 summarizes the ADDR process that supports an API design first
approach.
The process achieves the following goals:
• Deliver an API design that emphasizes and solves the customer problems using
a vocabulary they understand.
• Reduce the constant design churn common with informal design processes.
• Optimize the entire organization, not just developers, for API design and
delivery.
• Avoid unnecessary steps whenever possible to expedite delivery.
• Create a repeatable process that delivers an API design with a mixture of
technical and nontechnical roles, some of whom don’t fully understand the
nuances of API design but are able to contribute their insights.
The Align-Define-Design-Refine Process 25
1. Identify
Digital
Capabilities
7. Document 2. Capture
the API Align Activity
Steps
5. High-Level
4. Model API
Design
Profiles
• Produce artifacts that may be referenced within the team and shared across the
organization rather than some scribbles on a whiteboard that fail to communi-
cate reason and intent about the resulting API design.
• Aligning and defining the APIs required to deliver desired outcomes based on
the jobs to be done by developers and end users (Chapters 3–6)
• Designing APIs that help meet the desired outcomes of the target audience
using the appropriate API styles along with common patterns and practices
(Chapters 7–9)
• Decomposing APIs into smaller services to shift complexity when needed
(Chapter 10)
• Improving the developer experience through a combination of robust documen-
tation, helper libraries, command-line interfaces, and testing strategy to ensure
consumers get up and running quickly and with confidence (Chapters 11–13)
26 Chapter 2 Collaborative API Design
3. Eric Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software (Boston: Addison-
Wesley, 2003).
4. Vaughn Vernon, Implementing Domain-Driven Design (Boston: Addison-Wesley, 2013).
API Design Involves Everyone 27
Developers design and write the code that makes it all work. Designers and user
experience (UX) experts pull everything together as a user interface, with an eye
toward usability.
Each person contributes their experience and can leverage their strengths and
skills as part of the API design process. For smaller organizations, a single person
may be required to fill multiple roles. Whenever possible, assign the more technical
roles separately from the product and business roles to ensure a healthy balance of
perspectives when designing APIs.
The roles typically involved in API design sessions may include, but are not limited
to, the following:
• API designers and architects help facilitate the design process and bring in
API design expertise.
• Subject matter experts (SMEs) and domain experts help to clarify require-
ments and shape the vocabulary used in API design.
• Technical leads are responsible for guiding implementation efforts and may
require additional clarifying questions for estimation purposes.
• Product managers incorporate market opportunities and customer needs into
the API design.
• Technical writers ask clarifying questions during scope and design sessions
that will impact the capabilities delivered and drive the production of API doc-
umentation and getting started guides.
• Scrum Masters and project managers provide input to assist in scheduling
and identifying risks.
• QA teams can provide input on designing testable APIs, determine how and
when to test Web APIs, and design test plans in parallel with development
efforts.
• Infrastructure and operations ensure network, server, container platforms,
message brokers, streaming platforms, and other necessary resources are avail-
able for the teams that are building and consuming APIs.
• Security teams review API designs for personally identifiable information
(PII) and nonpublic information (NPI) concerns, identify risks, limit the
surface area of attacks, and help to design APIs that will access sensitive
data.
An API design process integrates perspectives from each of these roles to align
the business with development teams, define the clear goals and outcomes of an
API, and design the API to meet the defined goals. Upcoming chapters explore this
process in further detail.
28 Chapter 2 Collaborative API Design
Summary
The design of an API’s contract is a separate and critical step of software delivery.
API design requires communication within the organization and with the developers
using the API. It helps to course-correct wrong assumptions. It also encourages
communication between business, product, and technology teams.
An API design-first approach takes an outside-in perspective on the design of
an API by focusing on the customers and developers who are building the solution.
Combined with design techniques that take a bottom-up approach, APIs will have
a more balanced design that both reflects the domain and the needs of customers
and developers. An API design process requires a variety of roles that help to align,
define, and design the capabilities and outcomes that APIs will provide.
With the introduction to the art and fundamentals of API design complete, it is
time to dive into the details of the ADDR process with the first phase: Align.
Part II
29
This page intentionally left blank
Chapter 3
When we buy a product, we essentially “hire” it to help us do a job. If it does the job
well, the next time we’re confronted with the same job, we tend to hire that product
again. And if it does a crummy job, we “fire” it and look for an alternative.
— Clayton M. Christensen, Taddy Hall, Karen Dillon, and
David S. Duncan
1. Identify
Digital
Capabilities
7. Document 2. Capture
the API Align Activity
Steps
5. High-Level
4. Model API
Design
Profiles
Figure 3.1 The Align phase begins with identifying digital capabilities.
31
32 Chapter 3 Identify Digital Capabilities
APIs are the most common manifestation of digital capabilities, as they power Web
and mobile apps, partner integrations, and workforce solutions. They allow the
casual or expert developer to take advantage of data, business processes, and internal
systems programmatically to produce desired outcomes. Organizations must develop
the skills to identify digital capabilities (see Figure 3.1) and use them to shape the API
design to help users produce results.
The ADDR process starts with defining the digital capabilities necessary to deliver
customer outcomes. It also elaborates on the specific activities and steps needed to
deliver the outcomes, prior to designing the APIs.
This chapter introduces the concepts of digital capabilities, explains how they
relate to APIs, and outlines an approachable method for mapping requirements into
a format that identifies necessary digital capabilities. These digital capabilities are
then used to inform the design of product and platform APIs.
managers, to name just a few. These roles represent the needs of the customers. An
effective API design includes input from many roles across the organization, not just
the technical details of how to push data in and out of a data store or legacy system.
It is also necessary to create alignment between stakeholders and the development
teams responsible for implementing the API. If the API lacks business context, it may
meet the needs of customers but lack sufficient factors to meet business goals. If the
API lacks a customer context, it may meet the needs of business but fail to deliver the
desired outcomes of customers. If it lacks both, the API will serve no real purpose and
efforts will have been wasted. In the ADDR process, digital capabilities are used to
create alignment between business, customers, and technology to avoid these negative
consequences.
In JTBD, jobs are more than just functions that need to be performed. Jobs are
really about the desired outcome or accomplishment. Jobs may be new and unsolved
or may be solved in some way that doesn’t quite meet the customers’ needs. A product
that produces the desired outcome is one that considers all of these factors about the
jobs to be done. JTBD applies to APIs as well as all other aspects of product and
software design with the organization.
The idea behind JTBD is rooted in the voice of the customer (VOC)3 from the
mid-1980s, where product managers attempted to improve product performance by
getting into the mindset of the customer. VOC combines market research data with
specific wants and needs that have been identified through surveys and customer
interviews.
Christensen also reminds us that there is an emotional and social side to the jobs
that a product attempts to solve. The job extends beyond the immediate problem to
include a reduction or removal of the anxiety involved. The product should provide a
positive experience while producing progress toward the desired outcome. Some may
even go so far as to offer enjoyment while completing the job.
3. Wikipedia, s.v. “Voice of the Customer,” last modified July 15, 2021, 12:112, https://en.wikipedia.org/
wiki/Voice_of_the_customer.
36 Chapter 3 Identify Digital Capabilities
Job stories were created by Alan Klement4 and are based on JTBD formulated by
Christensen. They offer a simple framework to capture all of the aspects of the job
to be done.
Teams producing job stories will find that their API designs focus more on the
desired outcomes of the audience. They will also have the details necessary to cre-
ate acceptance criteria for automated tests. It is important to note that job stories
shouldn’t contain implementation specifics. Instead, they should elaborate on what
needs to happen to make the necessary progress to deliver the outcome.
The ADDR process leans heavily on job stories to capture business requirements in
a customer-centric way. Job stories express customer requirements in a simple format
and provide a natural way to identify digital capabilities that will drive API design.
Figure 3.2 shows an example Forgot Password job story, highlighting its three
components.
The example job story in Figure 3.2 demonstrates how a job story may be used to
inform the design of a digital capability. In this case, it captures a digital capability
titled Reset My Password. This is one of many digital capabilities the API must offer
to meet the needs of the target customers.
4. Alan Klement, “Replacing the User Story with the Job Story,” JTBD.info, November 12, 2013, https://
jtbd.info/replacing-the-user-story-with-the-job-story-af7cdee10c27.
Writing Job Stories for APIs 37
The triggering event or situation When I can't recall my password for an account
that I've logged into successfully in the past
The capability required I want to reset my password to something new
The outcome or goal desired So I can login successfully to the application
once again
• What is the desired outcome that the customer wish to experience to solve the
problem?
• What is the job required to achieve the outcome?
• Given these two answers, does the original problem best describe the trig-
gering situation, or is there a better way to express the problem in job story
format?
not be sure why they need it. Use the following questions to guide the discussion and
help to formulate job stories based on their desired outcomes:
• What is the problem, as described by the customer, that drives the desired
outcome?
• What is the job required to achieve the outcome? If multiple tasks are identified,
summarize them into a single job description.
• Does the desired outcome still best express their need, or should it be rewritten?
track of specific details that have been previously discussed. Consider the following
example that has too many details:
When a job story contains too many details, extract the details as additional
items below the job story. Doing so ensures the details are not lost and keeps the job
story clear and focused. Here is the same job story, rewritten with the details moved
outside of the job story narrative:
These details can be extracted into bullet points in a document or Markdown file
or added as an additional notes column in a spreadsheet.
Consider adjusting job stories that contain features into a standard job story
structure. If the team is concerned about losing details about the feature in the job
story, move feature details into an “additional details” section of the job story. The
feature details can then be referenced at a later point in the design process.
For example:
This approach provides a nice blend between job stories and user stories.
recommendations, but feel free to use anything that enables communication and
collaboration within and across teams.
• Spreadsheets: Spreadsheets are the universal tool and are quite useful for cap-
turing job stories. One job story per row in a spreadsheet will suffice. The first
column should be a job story identifier. Dedicate the next columns to each of
the three components, “When,” “I want to,” “so I can.” Finally, add a fifth
column for notes. Many spreadsheets support collaborative editing, enabling
multiple people to review, comment, and contribute as needed.
• Documents: Documents are also useful, though they are a bit less structured.
They are useful when teams wish to mimic an index card style for capturing
job stories. Start with a heading that indicates the job story identifier, such
as a number or brief description. Place each of the three components of a job
story, “When,” “I want to,” “so I can,” on a separate line for readability. Leave
room for capturing additional insights or details as a list of bulleted items. Add
a blank space between each job story to help separate each one, or assign one
job story per page.
• Markdown files: Markdown is a text file with an approachable syntax use-
ful for capturing job stories. Markdown files may be used to export job sto-
ries into HTML, PDF, and other formats. Use a single Markdown file with
all job stories, or create a Markdown file for each job story. Combine with
a version control system, such as git, to view a history of changes to the job
stories. Of course, this approach is targeted at teams with deeper technical
expertise.
Summary
APIs are digital capabilities that help turn desired outcomes into reality through
automation. An API designed with these outcomes will help to deliver a better API
design for the target audience.
5. https://bit.ly/align-define-design-examples
Summary 43
Job stories offer contextual understanding of the desired outcomes and the digital
capabilities that will be necessary to make them a reality. Through the process of
composing job stories, a shared understanding of business and customer needs for
all stakeholders is established. The more effort that is placed into composing job
stories, the more likely the API will meet the needs of customers. Job stories are the
first artifact needed to align all stakeholders prior to API design. The next chapter
discusses how to expand job stories into the activities and activity steps that will be
the foundation for API design.
This page intentionally left blank
Chapter 4
The real story is that software developers are spending a relevant amount of
their time learning, in order to do things they don’t have a clue about. Differ-
ently from other professions, we’re doing things for the first time, most of the
time (even if it looks like the same old typing from the outside).
— Alberto Brandolini
1. Identify
Digital
Capabilities
7. Document 2. Capture
the API Align Activity
Steps
5. High-Level
4. Model API
Design
Profiles
Figure 4.1 The next step in the Align phase is to capture activity steps.
45
46 Chapter 4 Capture Activities and Steps
This quote from Alberto Brandolini resonates with many teams that are faced with
building software in less familiar domains. While some developers can stay in the
same business vertical for most or all of their career, most do not have that luxury.
Developers are required to understand a new domain quickly, translate it into soft-
ware, and repeat throughout their career. They must quickly become familiar with a
domain such that they are able to turn it into working software that includes user
interfaces, APIs, and data models.
The ADDR process helps to bridge this gap through a series of rapid design
steps. Chapter 3, “Identify Digital Capabilities,” detailed the first step of the API
design process through the understanding of desired outcomes. The next step is
the gathering of details from stakeholders, development teams, and business
domain experts to better understand the concepts, processes, and workflows of the
domain.
This chapter addresses how to capture domain details and expected behavior
using an activity-based structure (see Figure 4.1). It also introduces the
EventStorming framework as a collaborative way to explore the domain. The result
is a deeper understanding of the domain, alignment between all team members,
and a foundation for defining and designing the APIs that will deliver the necessary
digital capabilities.
During this part of the process, all team members gain deeper understanding and
alignment on the solution. If requirements are vague or uncertainty remains, the
team may choose to proceed into a collaborative EventStorming session to explore
the solution further. EventStorming is detailed later in this chapter.
Table 4.1 Example Activities for JSON’s Bookstore Place an Order Job Story
Digital Capability Activity Participants Description
Place an Order Browse for Books Customer Browse or search for books
Place an Order Shop for Books Customer, Call Center A customer adds books to a cart
Place an Order Create an Order Customer, Call Center A customer places the order using
the contents of the shopping cart
48 Chapter 4 Capture Activities and Steps
Table 4.2 decomposes the activities for JSON’s Bookstore into activity steps.
Notice that some activities may have only a single step, whereas others may have
multiple steps. This is common, as some activities are more complex than others.
Repeat this process for each job story. Review the activities and steps with SMEs to
gain feedback and to ensure proper alignment. Once completed, proceed to the Define
phase detailed in Chapter 5. If requirements are not clear enough to produce activities
and steps, more work will need to be done. The API workshop examples,1 available
on GitHub, provide templates and examples for capturing job story activities.
1. https://bit.ly/align-define-design-examples
Using EventStorming for Collaborative Understanding 49
CASE STUDY
EventStorming for International Wire Transfers
2. https://www.eventstorming.com
50 Chapter 4 Capture Activities and Steps
The most valuable insight, however, was the number of unknowns around
the specifics of currency conversion. No one was familiar with the internal
policies regarding when a currency conversion was conducted. There were
several options, from performing the conversion at the time of wire initiation
to waiting until the wire transfer process started. The gap in knowledge was
identified within an hour of starting the session. It was decided that further
investigation was necessary. Domain experts were brought into the session to
clarify the matter.
With better knowledge at hand, some significant decisions needed to be
made. The session was halted to gain further clarification on the scope of
the release. The EventStorming was concluded at a future time when more
information was known, ensuring that the initial release met all business and
customer needs.
Had the EventStorming session not been conducted, developers would
have assumed a specific set of business policies with regard to currency con-
version. SMEs would have required a different set of business policies, forc-
ing developers to make last minute changes and incur significant technical
debt to deliver on time.
The EventStorming process starts by identifying business domain events for a job
story or group of job stories. Everyone captures these events on stickies, all of the
same color (typically orange), and places them on the canvas. (Brandolini recommends
using consistent colors for specific items. See color list under “Step 4: Expand Domain
Understanding.”)
Domain events are phrased in the past tense to indicate that something has
already happened. Phrasing domain events in the past tense can be challenging for
some attendees. Help them rephrase the domain events until the habit is built. Being
consistent in this effort pays off during subsequent steps. Table 4.3 demonstrates the
preferred naming conventions for domain events and those to avoid.
This step should offer two passes of 15 to 30 minutes each. For a session with a
larger scope, more passes may be required. The result is a large number of unordered
sticky notes scattered all over the canvas.
As events are placed, attendees may start to slow down. This is common and
easily remedied. Between each pass, review some areas of the canvas to identify
missing domain events. Ask attendees to review all domain events and identify
causation events that may come before a business domain event. If the causation
event is missing, have them add it as a new domain event sticky note.
Figure 4.2 demonstrates what this would look like when capturing the business
domain events for JSON’s Bookstore job stories 1 and 4 captured in Chapter 3,
Table 3.2.
Once the session is complete, take a brief break, then proceed to the next step.
Next, the domain event stickies are ordered into a narrative from beginning to end.
Along the way, duplicate events are removed, and clarifications are made to ensure
the events start to frame the narrative.
Search
Results
Displayed Book
Removed
from Cart Recent
Recent Books
Books Requested
Book
Listed Search
Issued Book Details
Displayed
Book
Added to Cart with Checkout
Cart Total Checkout Completed
Displayed Started
Valid
Payment
Received
Figure 4.2 An example of domain event sticky notes for the JSON’s Bookstore online store.
The events have now been captured and will be organized in the next step.
The facilitator is responsible for asking the group clarifying questions to ensure
the narrative is composed properly. Find the starting domain event for the narrative,
then seek to find the next domain event and place it after the first. Leave plenty of
space on the canvas to insert domain events as needed.
It is common for sessions to become stuck if there are branching or parallel
narratives. To help expedite the session, select a single narrative and order the
domain events accordingly. Branching or parallel narratives may be captured below
the primary narrative, if desired.
Figure 4.3 illustrates creating a narrative from the business domain events previ-
ously identified.
While this step may appear to require very little time, expect conversations to
emerge as the narrative is established. Therefore, allocate between one and two
hours minimum. If necessary, turn some of the domain events to a 45-degree angle
to note that they need to be revisited and proceed with the remainder of the narra-
tive. Once the overall narrative is established, revisit the domain events in question
or mark them with hotspot (typically hot pink) stickies for follow-up.
List Recently Added Books + Search for a Book Place an Order
Recent Recent Book Book Search Book Book Checkout Cart with Valid Checkout
Books Books Details Search Results Added to Removed Started Total Payment Completed
Requested Listed Displayed Issued Displayed Cart from Cart Displayed Received
Figure 4.3 Business domain event sticky notes arranged into a linear narrative for JSON’s Bookstore.
How EventStorming Works
53
54 Chapter 4 Capture Activities and Steps
Once all events have been cleaned up and grouped in a general timeline, the group
seeks to ensure that no events are missing. To do this, the group starts to walk from
left to right to tell the full narrative. If events are missing or need clarification, they
are changed immediately. It is at this step that a large surface area is beneficial to
ensure sticky notes may be moved around and for filling in gaps in the narrative.
It is also at this step that all domain concepts need to be unified to establish a
common vocabulary. This vocabulary will evolve into the ubiquitous language for
each bounded context that will be identified in the next step of the ADDR process.
Figure 4.4 demonstrates cards that may be attached to the EventStorming canvas to
unify common vocabulary. If necessary, rewrite existing domain events to use the new
vocabulary. It won’t take long for the group to start adopting the new terminology.
Expect this step to take at least an hour as questions are raised and discussions
emerge.
After the events have been ordered, additional sticky note colors are used to expand
domain understanding. Figure 4.5 illustrates a portion of the Place Order job story
for JSON’s Bookstore using these additional types of sticky notes.
Book Cart
Figure 4.4 Two examples of cards used to capture vocabulary clarifications during a
session. These cards will become part of the ubiquitous language for a bounded context
identified in the next step of the process.
How EventStorming Works 55
Use third
party?
<Hotspot>
<Aggregate>
Book
Remove Book Shopping
Shopper Removed
from Cart Cart
from Cart
<Aggregate>
Figure 4.5 The Place Order job story for JSON’s Bookstore, expanded to include additional
color-coded sticky notes for commands, aggregates, and users. There is also a hotspot sticky
with an open question to resolve after the session.
Following is a summary of the common sticky note types—and the colors typi-
cally designated for each type—that are used throughout an EventStorming session:
• Business event (orange): The result of an action or policy that indicates for-
ward progression through a workflow or process
• Hotspot (bright pink): Unknown or missing data that requires research and
follow-up after the session.
• Command (dark blue): An action taken by a user or system.
• Aggregate (large, pale yellow): Behavior or logic that executes as a result of
a command and behavior and often results in one or more events. In domain-
driven design (DDD), Aggregates are defined as units of transactional consist-
ency. In EventStorming, this is a higher-level representation of workflows, state
machines, and other behavior.
• Policy (lilac): The triggering event or motivation for why a new command is
executed and is always required. It acts as the bridge or glue between an event
and a command. Policies may start with the word when or whenever.
56 Chapter 4 Capture Activities and Steps
• External system (pale pink): Systems outside of the solution. These may be
external to the team but internal to the organization or a third-party system.
They are best thought of as aggregates outside of the group’s control.
• User interface (white): A user interface that will offer one or more roles the
capability to execute a command against an aggregate.
• User (yellow, small): A specific role that is interacting with the system,
typically via a UI, but also perhaps as a result of an automated call, email, or
other mechanism.
Start by adding commands (blue stickies) to capture what actions are taken by
a system or user that will result in one or more of the identified business domain
events. Commands are sent to aggregates, so capture those as well. Business
rules, often phrased as “when xyz happens, . . .” may be captured as policies
(lilac). Hot pink stickies are used as hotspot indicators where more information
is required.
Finally, the narrative is reviewed from both directions, start to end and end to start,
to ensure all elements have been captured. Important events and triggers are clearly
marked to denote key transitions between steps. Figure 4.6 shows a fully explored
Place an Order job story using the sticky notes necessary to express the understanding
gained during the session.
This completes the EventStorming session. The canvas should be saved for
future reference, as it will be useful for informing future steps in the API design
process. Consider taking photos of the canvas and sharing them with the team.
Rolling the canvas up or relocating it to a shared space is helpful for team mem-
bers colocated in the same office space. If the team used a digital tool, such as
Miro,3 export the work as a PDF or image for sharing on a wiki or as part of
other project assets.
Finally, write the activities and activity steps identified during the session using
the format outlined previously.
3. https://miro.com
How EventStorming Works 57
<Aggregate>
Book
Remove Book Shopping
Shopper Removed
from Cart Cart
from Cart
<Aggregate>
Order Create
Shopper Checkout Order from
Management
Cart
<Aggregate>
Order with
Checkout Order Details Order Details
Total
Started Model Screen
Displayed
Which
payment
service?
<Hotspot>
Submit Valid
Shopper Payment
Payment Payment
System
Details Received
Payment Payment
Checkout
Details Success
Completed
Model Screen
Figure 4.6 A completed EventStorming canvas for the JSON’s Bookstore Place Order
job story. Because of space limitations, the single line of sticky notes is wrapped into two
additional lanes.
58 Chapter 4 Capture Activities and Steps
There are also circumstances when the value of an EventStorming session may be
reduced. To avoid spending unnecessary time in an EventStorming session, consider
these factors that contribute to an ineffective session:
• The business process is well known and documented, as results will likely pro-
duce the same insights.
• The scope of the problem is small enough that the identified business require-
ments are sufficient and complete.
• The business requirements have not yet been identified. In this case, start with
constructing job stories, covered in Chapter 3, to clearly define the desired out-
comes and parties involved.
• Business stakeholders cannot attend or do not see value in the exercise. While
development teams may still conduct a session, doing so may lead to modeling a
process based on too many technical assumptions that do not meet business needs
• Software delivery has begun, and delivery dates are fixed. If the teams are early
enough in the delivery process, the output of the session may be used to make
software architectural and design adjustments prior to a release. Otherwise,
teams will be forced to proceed with existing decisions rather than the insights
obtained through an EventStorming session
session, so they won’t know the “proper colors” anyway. If experienced participants
complain about the wrong colors, request that they be in charge of the supplies for
any future sessions.
A large wall is also required for hanging a large paper sheet as a modeling surface
where sticky notes are applied and moved around as needed. Some organizations
prefer to use paper sheets that are no more than 18 to 24 inches (45–60 cm) to allow
two or three lanes of sheets along the wall.
A legend should be visible to ensure that everyone is reminded of each color of
sticky note used. Be sure to write out a legend that shows all sticky note colors, their
type (e.g., “Business Domain Event” for orange stickies), and arrows that show how
they are typically combined to produce a narrative. An example legend is shown in
Figure 4.7.
Black markers should be used to write on the sticky notes, ensuring words are
easy to read when stepping away from the modeling surface. Ensure there is at least
one marker for each participant, with a few extra scattered around the room in case
some are lost.
Remote sessions may choose to use a diagramming tool to simulate sticky notes.
Another option is to use a shared document that is accessible by each attendee,
applying similar color-coding to text as those used with sticky notes. Whatever is
chosen, practice using the tool prior to the event to ensure an effective use of time
during the session.
Policy
Hotspot
Payment
System
<External>
Figure 4.7 A legend that shows how common color-coded sticky notes work together to
capture the domain during an EventStorming session.
62 Chapter 4 Capture Activities and Steps
At session kickoff, review the items once more. Reviewing everything the day of
the session helps to bring everything top-of-mind and sets the expectations for the
session.
Facilitating an EventStorming Session 63
A second email from the facilitator should be sent two weeks after the
session. This email should ask how the output of the session is being used by the
team. Use this opportunity to find blocking issues that prevent the team from
proceeding. Schedule a follow-up discussion to coach the team on next steps if
they are blocked.
Finally, consider writing up a case study of the session and including quotes
from attendees, if permission is provided. This helps teams uncertain about the
EventStorming process understand the value that it provides and increase their
willingness to invest the necessary time. It also helps to share team wins across the
organizations.
• The three-lane approach: Rather than a single sheet of paper covering the
wall, using narrow paper that allows for creating three separate lanes in paral-
lel. All initial events are attached to the top lane during business event iden-
tification. When ordering them into a narrative, the events are moved to the
middle lane. The top lane is then used to expand the available space when
expanding the canvas with new sticky notes beyond the initial business events.
The bottom lane is used as a “parking lot” for events considered out of scope
or identified as duplicate. This approach was built out of necessity, as the only
available paper rolls from an organization’s supply closet were only 12 inches
(30 cm) high.
• 45-degree angle sticky notes: When a note isn’t clear, needs to be rewritten
into past tense, or is revisited for another reason, anyone is empowered to tilt
a sticky note at a 45-degree angle. This flags the note for follow-up before pro-
ceeding to the next step of the process.
• Multipart EventStorming sessions: With the introduction of remote sessions,
fatigue often sets in more quickly than with high-energy in-person sessions. In
this case, consider breaking an EventStorming session into multiple parts that
take no more than two hours before a minimum of a one-hour break. If neces-
sary, a session may be spread across two days as long as all attendees will be
available to participate both days.
Summary 65
Summary
It is essential for teams to establish a detailed understanding of domain concepts,
processes, and workflows. Capturing these details as activities and activity steps
helps to align all team members and establish a foundation for future API design
work. The EventStorming framework may be used as a collaborative method for
exploring the domain concepts in detail alongside domain and subject matter
experts.
The next step in the API design process is to start defining the bounded contexts
and APIs needed to realize the digital capabilities offered by an API product or API
platform.
This page intentionally left blank
Part III
67
This page intentionally left blank
Chapter 5
Total unification of the domain model for a large system will not be feasible
or cost-effective.
—Eric Evans, Domain-Driven Design
1. Identify
Digital
Capabilities
7. Document 2. Capture
the API Align Activity
Steps
5. High-Level
4. Model API
Design
Profiles
Figure 5.1 The first step in the Define phase is to identify API boundaries.
69
70 Chapter 5 Identifying API Boundaries
Every API provides a mental model for how developers will integrate with it to pro-
duce the desired outcomes. Establishing the scope and responsibilities of an API
also helps guide the design of this mental model, contributing to a more positive
developer experience. By borrowing techniques from domain-driven design (DDD)
used in identifying bounded contexts, candidate APIs are identified and the respon-
sibility of each API is clearly defined from the artifacts produced in the Align phase
(see Figure 5.1).
Output from activity step identification, as detailed in Chapter 4, “Capture Activities
and Steps,” is useful in finding boundaries and therefore candidate APIs. These
candidate APIs realize the digital capabilities that will produce the desired outcomes
captured in job stories, as detailed in Chapter 3, “Identify Digital Capabilities.” Before
getting started with the process, it is important to understand how some mistakes in
defining API boundaries can lead to a poor developer experience.
Creating a single Books API is likely not the best route to clarity or sanity. API
changes would be constant as new contexts around the term books are introduced.
The result would be operations that mix and match the term in different ways until
the API becomes a confusing mess. Not only does this lead to a poor developer expe-
rience, it also contributes to significant delays when delivering new enhancements in
the future.
In larger organizations, a single team may become responsible for most of the API
through this incorrect assumption that a single Books API is the best way to organize
support across the book catalog, inventory, shopping, and fulfillment processes of
the organization. The velocity of delivery is greatly reduced as the remainder of the
organization waits for the single team to add the operations needed to support new
functionality.
Instead, seek to clarify single-word terms such as book with more context. For
example, a Books Catalog API helps to limit the scope of API operations to those
involved to catalog management. Within this clearly defined scope are additional
responsibilities, including managing public descriptions, associated author metadata
to each book, book covers, sample chapters, and more. Only those interested in cata-
log management need to work with this API.
<Aggregate>
Book
Remove Book Shopping
Shopper Removed
from Cart Cart
from Cart
<Aggregate>
Order Create
Shopper Checkout Order from
Management
Cart
<Aggregate>
Order with
Checkout Order Details Order Details
Total
Started Model Screen
Displayed
Which
payment
service?
<Hotspot>
Submit Valid
Shopper Payment
Payment Payment
Management
Details Received
<Aggregate>
Payment Payment
Checkout
Details Success
Completed
Model Screen
Figure 5.2 An EventStorming canvas is helpful in finding shifts in language, leading to the
identification of bounded contexts.
Naming and Scoping APIs 75
Shopping
API
<Aggregate>
Book
Remove Book Shopping
Shopper Removed
from Cart Cart
from Cart
<Aggregate>
Order
Creation
API
Order Create
Shopper Checkout Order from
Management
Cart
<Aggregate>
Figure 5.3 APIs identified through the shift in language on the EventStorming canvas.
As an example, the activities and activity steps from Table 4.2 in the previous
chapter are presented in Table 5.1. Notice the shift from Books to Cart, then to
Order and Payment. These shifts provide a starting point for identifying boundaries,
from which APIs are formed.
Order with
Checkout Order Details Order Details
Total
Started Model Screen
Displayed
Payment Which
Processing payment
API service?
<Hotspot>
Submit Valid
Shopper Payment
Payment Payment
Management
Details Received
<Aggregate>
Payment Payment
Checkout
Details Success
Completed
Model Screen
Table 5.1 Activity Steps for JSON’s Bookstore; Separators Indicate Shifts in Vocabulary That
Identify Boundaries
Digital Activity Activity Step Participants Description
Capability
Place an Order Browse for Books List Books Customer, List books by category or
Call Center release date
Place an Order Browse for Books Search for Customer, Search for books by author,
Books Call Center title
Place an Order Browse for Books View Book Customer, View the details of a book
Details Call Center
Place an Order Shop for Books Add Books Customer, Add a book to the customer’s
to Cart Call Center cart
Place an Order Shop for Books Remove Books Customer, Remove a book from the
from Cart Call Center customer’s cart
Place an Order Shop for Books Clear Cart Customer, Remove all books from the
Call Center customer’s cart
Place an Order Shop for Books View Cart Customer, View the current cart and
Call Center total
Place an Order Create an Order Checkout Customer, Create an order from the
Call Center contents of the cart
Place an Order Create an Order Pay for Order Customer, Accept and process payment
Call Center for the order
Naming and Scoping APIs 77
Avoid using the terms service and manager, as they are generally not useful in
understanding the purpose of the API.
The APIs in Figure 5.3 are named Shopping API, Order Creation API, and
Payment Processing API. This is a good start and clearly articulates the scope and
responsibility of each API.
Note
Some API designers may prefer to combine the Order Creation and Payment Pro-
cessing APIs, as they could be considered cohesive and therefore should exist as a
single API. They are separated in this simple example for instructional purposes.
However, field insights dictate that separating order creation and payment through
clearly defined boundaries allows for more complex payment processing at a future
time without burdening the order creation boundary with the added complexity.
Finally, separate the activity steps into the corresponding API based on the
boundary it represents. Table 5.2 captures the Shopping API, including the activity
steps relevant to the API.
Table 5.3 captures the checkout process for the Place an Order digital capability.
Finally, Table 5.4 captures the payment step as part of the Place an Order digital
capability.
Table 5.2 Shopping API, Discovered through Boundary Identification, with Corresponding
Activity Steps from JSON’s Bookstore
Digital Activity Activity Step Participants Description
Capability
Place an Order Browse for Books List Books Customer, List books by category or
Call Center release date
Place an Order Browse for Books Search for Customer, Search for books by author,
Books Call Center title
Place an Order Browse for Books View Book Customer, View the details of a book
Details Call Center
Place an Order Shop for Books Add Books to Customer, Add a book to the
Cart Call Center customer’s cart
Place an Order Shop for Books Remove Books Customer, Remove a book from the
from Cart Call Center customer’s cart
Place an Order Shop for Books Clear Cart Customer, Remove all books from the
Call Center customer’s cart
Place an Order Shop for Books View Cart Customer, View the current cart and
Call Center total
Place an Order Create an Order Checkout Customer, Create an order from the
Call Center contents of the cart
Place an Order Create an Order Pay for Order Customer, Accept and process payment
Call Center for the order
78 Chapter 5 Identifying API Boundaries
Table 5.3 Order Creation API, Discovered through Boundary Identification, with
Corresponding Activity Steps from JSON’s Bookstore
Digital Activity Activity Step Participants Description
Capability
Place an Order Create an Order Checkout Customer, Create an order
Call Center from the contents
of the cart
Table 5.4 Payment Processing API, Discovered through Boundary Identification, with
Corresponding Activity Steps from JSON’s Bookstore
Digital Activity Activity Step Participants Description
Capability
Place an Order Create an Order Pay for Order Customer, Accept and
Call Center process payment
for the order
With the boundaries clearly defined, API modeling can begin. This effort leads to
API profiles that define the operations and events each API will offer. API modeling
is detailed in the next chapter.
Summary
APIs benefit from careful scoping and assignment of responsibilities. Applying
boundaries helps to identify one or more APIs that will be required to deliver the desired
outcomes captured as job stories. This prepares the way for the next step, API modeling,
in which a blueprint of the API is formed and the foundation is laid for API design.
Chapter 6
API Modeling
1. Identify
Digital
Capabilities
7. Document 2. Capture
the API Align Activity
Steps
5. High-Level
4. Model API
Design
Profiles
Figure 6.1 The final step of the Define phase is to create API profiles in preparation for
transitioning to the Design phase.
79
80 Chapter 6 API Modeling
Developers are often tempted to start writing code immediately. Code is the primary
tool for developers. It is everything—the hammer, screwdriver, ruler, and saw. When
code is seen as the one and only tool to design an API, the quality of the API design
can suffer. The march to produce code for production becomes more valued than the
outcomes the API is meant to produce.
Of course, code produces value when it is used to explore a specific area of
a solution to reduce risk. It is also valuable to use code to experiment, surface
unknowns, or explore a new technology. The term tracer bullet was applied
to software by David Thomas and Andrew Hunt in their book The Pragmatic
Programmer.1 The term describes the use of code as a means of exploration and
risk reduction. Tracer bullet code delivers value through learning rather than
through production-ready code.
API modeling is a tracer bullet for API design. It is a technique for exploring the
necessary elements of an API prior to the design and delivery process. API modeling
(see Figure 6.1) helps to bring together the insights and artifacts from the previous
steps into an API profile that describes the scope and intent of the APIs needed to
deliver the desired outcomes of end users.
1. David Thomas and Andrew Hunt, The Pragmatic Programmer: Your Journey to Mastery, 20th
Anniversary Edition, 2nd ed. (Boston: Addison-Wesley, 2020).
The API Modeling Process 81
inform the design of an API that uses a combination of these API styles to support
the various digital channels for customers and partner integration needs.
listThingies() List/search for thingies Customer, Thingy Thingies.Listed Request Parameters: vendorId, …
Shopper Returns: Thingy[ ]
… … … … …
Those who have produced the artifacts from the previous Align-Define-Design-
Refine (ADDR) process steps will typically complete the API modeling process in
under two hours. For those who skipped some of the steps, API modeling may take
several hours to complete.
Figure 6.3 JSON’s Bookstore’s Shopping API with a name, description, scope, and operation names filled in using the details from the bounded
context exercise in Chapter 5, “Bounded Contexts in API Design.”
The API Modeling Process 85
While it may be tempting to use a database schema as the starting point for
resource identification, keep in mind that API designs should not leak internal
implementation details. A database schema reflects optimizations for trans-
actional read and write operations rather than exposing business domain
concepts over a network API.
It is best to work top-down when modeling an API to avoid leaking inter-
nal data model decisions into the API design. If the implementation phase
demonstrates a one-to-one relationship between resources and a database
schema, then it becomes a “happy accident,” as painter Bob Ross2 would say.
2. Bob Ross, “We Don’t Make Mistakes,” clip from The Joy of Painting, season 3, episode 5, “Distant Hills”
(Schmidt1942, 2013; originally aired Feb. 1, 1984), https://www.youtube.com/watch?v=wCsO56kWwTc.
86
Chapter 6
Figure 6.4 Identifying the Book and Cart resources for JSON’s Bookstore’s Shopping API profile.
The API Modeling Process 87
Figure 6.5 Capturing each resource for the Shopping API, along with some basic details
about each one as a starting point.
Figure 6.6 Reviewing the resource relationships introduces a challenge: where does the quantity
of books belong?
Figure 6.6 shows the resources identified for the Shopping API, along with pos-
sible relationships. The relationship between a Book resource and Book Author
resource is independent. There may be a reference between them, but each one exists
independently of the other.
Notice in Figure 6.6 that a question needs to be addressed regarding where the
quantity is specified when a book is added to a cart. This issue needs to be explored
further. When a book is added to a cart, additional details that may be important include
the quantity and the price of each book. This indicates an associative relationship that
requires a new resource, in this case a Cart Item. The result is shown in Figure 6.7.
The operation addB ook ToC art( ) is renamed addI temToC art( ) , and remov eB ook -
F romC art( ) is renamed remov eI temF romC art( ) to reflect the introduction of the Cart
Item resource, as shown in Figure 6.8.
title The book title book The books currently in cartItems The items currently in
Independent the cart for purchase Dependent the cart for purchase
isbn The unique ISBN
of the book qty The quantity of the item subtotal The total cost of
in the cart (default 1) all books in the cart
authors List of Book
Author resources unitPrice The unit price salesTax The sales tax to be
represented as a whole applied
number. For example,
$1.99 USD would be vatTax Any value-added tax to
Independent 199 be applied
Figure 6.7 A new resource, Cart Item, is added because of the associative relationship between Book and Cart.
The API Modeling Process
89
90
Chapter 6
Figure 6.8 The Shopping API profile, revised to reflect the introduction of the Cart Item resource.
The API Modeling Process 91
Event names should be presented in past tense and should apply the preferred
standards and practices used within the organization. Figure 6.9 expands the previ-
ous model with events that each operation will emit. Notice the past-tense naming in
the Emitted Events column.
The API profile now reflects the events that the identified operations will emit.
Some operations may emit only one event, some more than one event, and some may
not need to emit any event at all.
1. Safe: The operation does not make any state changes to the target resource(s).
This safety classification is assigned to all read-based (GET) operations.
2. Idempotent: The operation makes state changes to the target resource(s),
but if the operation is executed with the same input, it will produce the same
result. This is important, as it informs API clients if they can reissue a request
that previously failed, without additional side effects. This safety classification
is assigned to replace and delete operations (PU T and D EL ETE).
92
Chapter 6
Figure 6.9 JSON’s Bookstore’s Shopping API, now with emitted events added for each operation in the API profile.
Validating the API Model with Sequence Diagrams 93
3. Unsafe: The operation makes state changes to the target resource(s) and can-
not guarantee the same results if called multiple times with the same input.
This safety classification is typically assigned to create and update (POS T and
PATC H) operations.
Review each operation to determine the safety classification type that is required
of the operation. Doing so during API modeling provides additional insights during
the design process.
Figure 6.10 shows the Shopping API example expanded to include input and
output details for each operation, synchronicity, and safety classification.
The API profile may then be finalized by capturing any architectural requirements,
such as SLAs, necessary to support consumers and industry standards that may need
to be considered during design (e.g., adherence to an open banking standard).
Refer to the API workshop examples repository4 available on GitHub for API pro-
file templates and examples to help jumpstart the effort.
4. https://bit.ly/align-define-design-examples
94
listBooks( ) List books by category Customer, Book, Book Author Books.Listed Request Parameters: categoryId, releaseDate
or release date Call Center Returns: Book[ ]
safe / synchronous
searchBooks( ) Search for books by Customer, Book Books.Searched Request Parameters: searchQuery
API Modeling
viewBook( ) View book details Customer, Book Book.Viewed Request Parameters: bookId
Call Center Returns: Book
safe / synchronous
addItemToCart( ) Add a book to the Customer, Cart Item, Cart Cart.ItemAdded Request Parameters: cartId, bookId, quantity
customer’s cart Call Center Returns: Cart
unsafe / synchronous
removeItemFromCart( ) Remove a book from Customer, Cart Item, Cart Cart.ItemRemoved Request Parameters: cartItemId
the customer’s cart Call Center Returns: Cart
idempotent / synchronous
clearCart( ) Remove all books from Customer, Cart Cart.Cleared Request Parameters: cartId
the customer’s cart Call Center Returns: Cart
safe / synchronous
viewCart( ) View the current cart Customer, Cart Cart.Viewed Request Parameters: cartId
and total Call Center Returns: Cart
safe / synchronous
Figure 6.10 JSON’s Bookstore Shopping API with operation details added to the API profile.
Evaluating API Priority and Reuse 95
listB ook s( )
B ook [ ]
C art
B ook [ ]
C art
v iew C art( )
C art
Order
PaymentD etails
Figure 6.11 JSON’s Bookstore’s Place an Order job story, represented as a sequence diagram
to ensure all operations were identified as part of the API modeling process to implement job
story outcomes and associated activities.
First, assess the business and competitive value that the API offers. Ask the
following questions to assess the value that each API brings:
• Does the API help provide a competitive advantage over other market offerings?
• Does the API reduce the cost of doing business, perhaps by reducing manual
processes?
• Does the API create a new revenue stream or improve an existing revenue
stream?
• Is the API producing business intelligence, market insights, or decisioning
factors?
• Does the API automate repetitive tasks that free the organization for more
critical business functions?
If the answer to all the questions is no, then the value produced is low. Answering
yes to one or more questions results in the API offering value to the business or
marketplace.
Next, size the effort to build each API from scratch. One approach is to use a
relative sizing approach by classifying the APIs as small, medium, or large. Consider
the details surfaced during EventStorming and write activity steps to roughly
estimate the size and complexity to build the API from the ground-up.
Finally, determine if there are existing APIs that could be leveraged or extended.
These APIs may be commercial off-the-shelf (COTS) APIs, internal APIs produced
by another team, or open-source solutions that could be leveraged to speed delivery.
Organizations often forget this step, resulting in wasted time and effort building
duplicate or noncore APIs. Performing this step encourages organizations to reuse
APIs first and build new APIs only when necessary.
Figure 6.12 shows a tabular format that captures examples of these details for
JSON’s Bookstore API profiles modeled in this chapter.
Summary
API modeling helps to bring together the insights and artifacts produced in the Align
phase of the design process into a model that describes the scope and intent of the
APIs required. Review the following checklist to ensure nothing was missed:
• Resource taxonomies identify the properties of each resource along with the
relationships and dependencies among them.
API Profile Business and In-House Existing Internal/Third-Party APIs
Competitive Value Build Effort
Shopping API Medium Medium Third-party eStore solutions (high complexity to
customize and add our internal recommendation engine support)
Order Creation API Medium Medium Third-party order processing APIs (may include fulfillment
support also)
Payment Processing API Small Large Various third-party payment processors
Figure 6.12 A sizing and prioritization effort for the current set of JSON’s Bookstore’s API profiles captured during API modeling.
Summary
97
98 Chapter 6 API Modeling
• API profiles are created to offer a high-level specification of each API, inde-
pendent of how the APIs will be designed and implemented.
• Sequence diagrams help to validate how the APIs deliver the outcomes cap-
tured in the job stories.
• Sizing and prioritization help to ensure that APIs are reused when possible
and built when necessary.
By spending time modeling APIs, teams can clearly articulate the needs of each
API. They are also able to identify the effort involved in building them from scratch
and to consider internal or third-party APIs that may meet requirements. Reusing
existing APIs saves weeks or months of delivery effort.
With API modeling complete, the Define phase of the ADDR process has reached
an end. The API Design phase is next. This phase migrates the API profiles into a
specific API style. Chapter 7 shows how to use API profiles to inform the design
of a REST-based API. If the target API style is not REST, feel free to jump to the
appropriate chapter.
Part IV
Designing APIs
Now that APIs have been modeled as part of the Define phase, the team has a better
understanding of the scope of each API and the operations it needs to support to
realize the desired outcomes. The next phase of the Align-Define-Design-Refine
(ADDR) process is to migrate the modeled API profiles into API designs.
There are many choices of API styles, from REST to RPC and beyond. This
section details a step-by-step process on producing a high-level design using the
artifacts created during the Align and Define phases of the process. Feel free to
jump to the specific chapter needed to apply the API style of choice.
99
This page intentionally left blank
Chapter 7
1. Identify
Digital
Capabilities
7. Document 2. Capture
the API Align Activity
Steps
5. High-Level
4. Model API
Design
Profiles
Figure 7.1 The Design phase offers several options for API styles. This chapter covers REST-
based API design.
101
102 Chapter 7 REST-Based API Design
As teams move from the modeling to the design phase, they are faced with a variety
of decisions. Some of these will resolve easily, while others will take time and delib-
eration. Just know that it is difficult to get an API design right the first time. There-
fore, teams are encouraged to spend time designing and prototyping their APIs to
gain feedback from early adopters before coding begins.
This chapter presents an overview of REST, along with a step-by-step process
for migrating an API profile, created during the API modeling phase, into a
REST-based API design. Along the way, various decisions and common design
patterns are explored. The result will be a high-level API design (Figure 7.1) that
applies REST-based principles.
Principle 3: Select the API design elements that match the need
Trying to find the perfect API style is a fruitless endeavor. Instead, seek to
understand and apply the API elements appropriate for the need, whether that
is REST, GraphQL, gRPC, or an emerging style just entering the industry. The
next three chapters provide insights into popular API styles to help teams select
the right style or styles that fit the need. The chapters discuss when to apply
each style, when to select synchronous or asynchronous interaction models, and
whether to offer SDK libraries.
1. Roy Thomas Fielding, “Architectural Styles and the Design of Network-based Software Architectures”
(PhD diss., University of California, 2000), https://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm.
What Is a REST-Based API? 103
The paper does not require the use of HTTP as the underlying protocol for
REST-based architecture. It does, however, discuss how the authors of the HTTP
specification relied on the outlined architectural constraints to make it more
evolvable. Because HTTP is the protocol of choice for most network-based APIs,
the details offered in this and later chapters rely on HTTP as the protocol of
choice.
The set of architectural properties outlined in Fielding’s paper serve to establish
constraints to create agreement around flexibility and evolvability when applied to
distributed systems, including network-based APIs:
• Client/server: The client and the server act independently, and the interaction
between them is only in the form of requests and responses.
• Stateless: The server does not remember anything about the user who uses
the API, so all necessary information to process the request must be pro-
vided by the client on each request. Note: This isn’t about storing server-
side state.
• Layered system: The client is agnostic as to how many layers, if any, there are
between the client and the actual server responding to the request. This is a key
principle of HTTP, allowing for client-side caching, caching servers, reverse
proxies, and authorization layering—all transparent to the client sending the
request.
• Cacheable: The server response must contain information about whether or
not the data is cacheable, allowing the client and any middleware servers to
cache data outside of the API server.
• Code on demand (optional): The client can request code from the server, usu-
ally in the form of a script or binary package, for client-side execution. This is
performed today by browsers requesting JavaScript files to extend the behavior
of a Web page. Code on demand is an opportunity for API teams to provide
JavaScript files for clients to retrieve that perform form validation and other
responsibilities. Thus, evolvability can be extended to clients through code on
demand.
• Uniform interface: Encourages independent evolvability by leaning on
resource-based identification, interaction using representations, self-descriptive
messages, and hypermedia controls.
REST Is Client/Server
The client/server architecture is an essential REST constraint. The server hosts
available resources, supporting operations through synchronous, message-based
interactions that use one or more representations as the client interacts with it.
Separating the client and server allows the user interface of the client to change
over time. New devices and interface styles may be used without requiring changes
to the server.
Most important is that the client is able to evolve independently of the server.
The server may offer new resources or additional representation formats without
negatively impacting the client. This is fundamental to why APIs may be offered as
products, with or without a preexisting user interface provided by the vendor.
REST Is Resource-Centric
The key abstraction of information in REST is a resource. As mentioned in Chapter 1,
“The Principles of API Design,” a resource consists of a unique name or identifier that
can reference documents, images, collections of other resources, or a digital
representation of anything in the real world, such as a person or thing.
What Is a REST-Based API? 105
Client
Client Cache
HTTP
CDN / EdgeCache
Load Balancer
HTTP
Logging
OAuth
Caching
Custom
HTTP
Figure 7.2 REST supports a layered architecture, allowing middleware servers to exist
between the client and server.
What Is a REST-Based API? 107
Hypermedia Controls
A hypermedia API is one driven by self-descriptive links in the representation.
These links point to other, related API operations that reference related resources.
They may also be used to reference other possible uses of the API, commonly
referred to as affordances. Fielding considered hypermedia important to a REST-
based architecture.
APIs that use hypermedia controls extend the conversation between client
and server by offering runtime discovery of operations. They may also be used to
convey server-side state to the client through the presence or absence of links in the
representation. This powerful concept is explored later in this section.
Hypermedia controls help connect the dots of various resources within and
across APIs, making it operate more like the Web. Imagine using a search engine
to find some results, only to never be offered links to click to explore the results.
Unfortunately, that is the way most design their APIs, offering only data and not
hypermedia controls that afford the client the opportunity to explore the depth of
the data and operations offered by the API.
108 Chapter 7 REST-Based API Design
{
"_ link s": {
"self ": {"h ref ": "/projects" },
"nex t": {"h ref ": "/projects? since= d26 6 f 6 cd& max R esu lts= 20" },
"prev ": {"h ref ": "/projects? since= 43b e8 07 d& max R esu lts= 20" },
"f irst": {"h ref ": "/projects? since= ef 2426 6 a& max R esu lts= 20" },
"last": {"h ref ": "/projects? since= 4e8 c7 4b e& max R esu lts= 20" },
}
}
API clients can be designed to use the nex t link to follow the search results page
by page until the next link is no longer present, indicating all search results have been
processed.
APIs that offer hypermedia controls help to create context-driven responses.
These controls are able to indicate what operations are possible, or not possible,
based on the presence or absence of hypermedia links. This capability avoids the
need to send Boolean fields or state-related fields that the client interprets to decide
what actions may be taken. Instead, the server determines this ahead of time and
conveys what can and cannot be done by the presence or absence of the links
provided.
What Is HATEOAS?
{
"articleI d":"12345",
"statu s":"draf t",
"_ link s": [
{ "rel":"self ", "u rl":"..."},
{ "rel":"u pdate", "u rl":"..."},
{ "rel":"su b mit", "u rl":"..."}
] ,
"au th ors": [ ... ] ,
...
}
Once the author is ready to submit the article for editorial review, the editor
would retrieve the article and receive the following actions based on the submitted
status of the article and the editor’s role:
{
"articleI d":"12345",
"statu s": "su b mitted",
"_ link s": [
{ "rel":"self ", "u rl":"..."},
{ "rel":"reject", "u rl":"..."},
{ "rel":"approv e", "u rl":"..."}
] ,
"au th ors": [ ... ] ,
...
}
Hypermedia controls have big implications for API-driven workflows that use
context-driven hypermedia controls. They help to reduce the amount of business
logic that has to be repeated in the client to mimic intended server behavior. Without
them, the client would need to be coded to know what actions are allowed based
on the status of the article and the role of the user. Instead, the client may be coded
to look for specific hypermedia links that indicate if specific buttons are displayed
or disabled for the end user, avoiding the need to keep in sync with the server-side
business logic. This ensures the API is evolvable without breaking client code.
110 Chapter 7 REST-Based API Design
It is important to note that any API style that does not encourage a unique URL
per resource is unable to take advantage of hypermedia controls. This is the case for
GraphQL and gRPC API styles, detailed in Chapter 8, “RPC- and Query-based API
Design,” along with older network API styles and messaging specifications such as
SOAP and XML-RPC.
The REST interface is designed to be efficient for large-grain hypermedia data transfer,
optimizing for the common case of the Web, but resulting in an interface that is not
optimal for other forms of architectural interaction.
While Fielding doesn’t define large grain explicitly, the Web is a good example. An
HTML page is sent as a single, complete resource. It is not split into separate assets
and retrieved separately. Once the HTML resource has been received and parsed, all
referenced images, JavaScript, and style sheets are retrieved individually.
REST is a good architectural style choice when the interaction level requires
course-grained resources over HTTP. This is common for APIs exposed over
the Internet that may encounter additional network latency or unpredictable
connectivity.
For finer-grained interaction styles, RPC or other styles may be a better fit. Some
RPC styles offer improved performance and long-running connections that do not
meet the REST constraints outlined by Fielding. This includes the choice of gRPC
and asynchronous API styles, discussed in Chapter 8 and Chapter 9, “Messaging,
Streaming, and Event-Based Async APIs,” for service-to-service and client-to-service
interactions.
2. Leonard Richardson, “What Have I Done?” (lecture, REST Fest, Greenville, SC, September 18, 2015).
112 Chapter 7 REST-Based API Design
In addition, many organizations default to REST for their API style choice when
offering customer- and partner-facing Web APIs. Most select the REST-based API
style internally because of the abundance of tooling, infrastructure, operational, and
management support. Because REST builds on the patterns of the Web, it is familiar
to developers, easily managed by ops teams, and offers an abundance of tools and
libraries for producing and consuming APIs.
Resource Path
Books
Book Authors
Carts
Cart Items
Figure 7.3 Begin by migrating the list of resources from the API profile created during
modeling into a preliminary API design. Note that Cart Items is a dependent relationship, as
detailed in Chapter 6, so it is indented slightly.
REST API Design Process 113
Dependent resources are nested under the parent, requiring the parent identifier
in the path to interact with the dependent resources.
The results are shown in Figure 7.4. Notice that the resource collections are plural
names. While not required, this convention is commonly found with REST APIs.
Migrate the list of operations, including their descriptions and request/response
details, into a new tabular format designed to capture the high-level design. This is
shown in Figure 7.5.
Resource Path
/books
/carts
/carts/{cartId}/Items
/authors
Figure 7.4 Convert each resource name into a URL-friendly name with dependent resources
nested under the parent.
114
Chapter 7
and total
Figure 7.5 Migrate the API profile’s list of operations, descriptions, and request/response details into a new design-centric tabular format.
REST API Design Process 115
• 200 codes indicate success, some with more clarity (e.g., 201 CREATED vs.
200 OK).
• 400 codes indicate a failure in the request that the client may wish to fix and
resubmit.
• 500 codes indicate a failure on the server that is not the fault of the client. The
client may attempt a retry at a future time, if appropriate.
Be sure to use the right code for the right reason. When in doubt, refer to the
most current RFC that details the intended usage for the code. If a specific code isn’t
available within the response code family, then use the default 200, 400, or 500 code
as appropriate.
Resource Path Operation Name HTTP Method Description Request Response
Figure 7.6 Using the list of paths identified earlier, assign the appropriate path and HTTP method to each operation based on the intended
usage.
REST API Design Process
117
118 Chapter 7 REST-Based API Design
Over the years, some strange decisions have been made by API designers.
One is the decision to use UNIX-style codes, where 0 indicates success and
1 through 127 indicate an error, for HTTP response code. Please do not invent
your own response codes. HTTP is designed to be layered, which means that
middleware servers that you don’t own might be involved between the client
and server. Creating your own codes will only cause problems with these
intermediary layers.
While the list of HTTP response codes is quite large, there are several that are
commonly used in API design. These are detailed in Table 7.3.
API clients should be prepared for any kind of response code, but it is not
necessary to capture every possible response code. Start by identifying at least one
success response code for each operation, along with any error codes that the API
may explicitly return. While the list of errors may not be comprehensive, start by
identifying the typical error codes that may be returned. Figure 7.7 shows the possible
success and error codes for the Shopping API.
Figure 7.7 Expanding the Shopping API design with success and error response codes captured in the Response column of the table.
REST API Design Process
119
120 Chapter 7 REST-Based API Design
Figure 7.8 Capturing the Shopping API design into the OpenAPI Specification v3, starting
with the name, description, and other important details.
3. https://swagger.io
REST API Design Process 121
Next, capture the details of each operation. For OAS v3, this begins with the path,
followed by each HTTP method supported at the path. It is also recommended to
add an operationI d property to each operation. Use the operation name from the
API profile, defined in Chapter 6. This makes the documentation process effortless
and helps to map the OAS description back to the API profile.
Using the details captured in the associated job stories created in Chapter 3,
“Identify Digital Capabilities,” write a short summary of the API to help readers
understand its purpose. Expand the details in the description field using the informa-
tion captured in the API profile in Chapter 6. Also, ensure all path parameters and
query arguments are captured. This is shown in Figure 7.9.
Figure 7.9 Expanding the Shopping API design documentation to include each operation.
122 Chapter 7 REST-Based API Design
Finally, capture all schema elements for resource representations in the schema
definitions section of the OAS v3 description. Use the resource models created dur-
ing API modeling, as described in Chapter 6. This is shown in Figure 7.10, where a
ListBooksResponse captures the response of the ListBooks operation.
Note in Figure 7.10 that the ListBooks operation returns an array of Book Sum-
mary instances that contain the basic details of each book in a search result. Adding
schema definitions that wrap an array response or that limit the acceptable proper-
ties for each operation’s request/response payload is often necessary. Operations that
Figure 7.10 Finalizing the Shopping API design documentation to include schema definitions.
REST API Design Process 123
Figure 7.11 Keep in mind that some operations may require custom schema definitions to
exclude specific fields not permitted for specific operations or to wrap search responses that
contain only summary details.
create or update a resource may also require separate schema definitions to prevent
read-only fields from being submitted. This is shown in Figure 7.11.
Use sequence diagrams to validate that the API design meets the needs captured
during the creation of job stories, EventStorming, and API modeling. Figure 7.12
shows a sequence diagram with a simplified form of HTTP for demonstrating
typical interaction patterns to produce the desired outcomes.
Once the API design has been captured in an API description format, the generated
documentation and sequence diagrams can be shared with others to obtain feedback
on the design. This is the final step in the API design process.
124 Chapter 7 REST-Based API Design
GET /b ook s
B ook [ ]
C art{cartid: 456 }
B ook [ ]
GET /carts/456
Figure 7.12 Use sequence diagramming to validate that the API design meets the needs
previously modeled.
implementations offer a chance for developers to experience how the API may work
using mock data. Tools are beginning to emerge that accept an API description
format, such as OAS, and generate a mock API implementation without writing a
single line of code.
Refer to Chapter 16, “Continuing the API Design Journey,” regarding other API
lifecycle techniques that help to gain feedback on an API, even after it has been
developed and deployed.
4. https://bit.ly/align-define-design-examples
126 Chapter 7 REST-Based API Design
Resource Serialization
The resource serialization category of representation formats is the most commonly
encountered. They directly map each property of the resource and its value into the
desired format, often in JSON, XML, or YAML. Binary formats such as Protocol
Buffers5 and Apache Avro6 are gaining acceptance as well.
These representation formats require explicit code that handles the serialization
between the resource and the target format. This mapping logic may be created
through code generators or hand coded. Formatter and representer libraries often
help to manage some of the mapping between the target format and an object or
struct that represents the resource in code.
No matter how serialization is handled, the parsing and mapping code is unique
to the resource, as it must be aware of the expected fields and any nested structures.
Listing 7.1 provides an example of a Book resource using a serialized representation
in JSON.
{
"b ook I d": "12345",
"isb n": "9 7 8 - 03218 3457 7 ",
"title": "I mplementing D omain- D riv en D esig n",
"description": "W ith I mplementing D omain- D riv en D esig n, V au g h n h as made an
important contrib u tion not only to th e literatu re of th e D omain- D riv en D esig n
commu nity, b u t also to th e literatu re of th e b roader enterprise application
arch itectu re f ield.",
"au th ors": [
{ "au th orI d": "7 6 5", "f u llName": "V au g h n V ernon" }
]
}
5. https://developers.google.com/protocol-buffers/docs/proto3
6. https://avro.apache.org/docs/current
Selecting a Representation Format 127
Hypermedia Serialization
The hypermedia serialization category is similar to resource serialization but adds
specifications on how hypermedia links are represented. It may also include specifi-
cations for including related and/or nested resources, called embedded resources, in a
uniform way.
Formats such as Hypertext Application Language (HAL)7 enable resource
serialization formats to be extended with hypermedia with few to no changes. This
prevents breaking existing API clients while migrating existing serialization-based
APIs to include hypermedia controls. This is why HAL tends to be a popular choice
when moving to hypermedia APIs. Listing 7.2 shows an example of a HAL-based
representation that extends Listing 7.1 with hypermedia links and a related list of
author resources.
{
"b ook I d": "12345",
"isb n": "9 7 8 - 03218 3457 7 ",
"title": "I mplementing D omain- D riv en D esig n",
"description": "W ith I mplementing D omain- D riv en D esig n, V au g h n h as made
an important contrib u tion not only to th e literatu re of th e D omain- D riv en
D esig n commu nity, b u t also to th e literatu re of th e b roader enterprise ap-
plication arch itectu re f ield.",
"_ link s": {
"self ": { "h ref ": "/b ook s/12345" }
},
"_ emb edded": {
"au th ors": [
{
"au th orI d": "7 6 5",
"f u llName": "V au g h n V ernon",
"_ link s": {
Not all hypermedia formats offer the same features. Mike Amundsen has created
a list of these factors, called H-Factors,8 that support reasoning about the level and
sophistication of hypermedia support across formats.
Hypermedia Messaging
Hypermedia messaging formats differ from serialization in that they propose a
uniform message-based format to capture resource properties, hypermedia controls,
and embedded resources. This makes it easy to use a single parser across all resources
represented by the message format rather than unique mapping code for each
resource type to parse a serialized format such as JSON or XML.
While the differences are nuanced between serialization and message-based
formats, consider that teams will no longer need to argue about what the JSON or
XML payload should look like. Instead, they focus on the resource representations,
relationships, and hypermedia controls within the message format itself. No more
meetings to decide if a data wrapper is required around a JSON-based response!
Hypermedia messaging formats include JSON:API9 and Siren.10 Both of these
formats offer a single structured message that is flexible enough to include simple
or complex resource representations and embedded resources, and both offer
hypermedia control support.
Siren’s messaging capabilities are similar to JSON:API’s, but it also adds metadata
that is useful for creating Web-based user interfaces with minimal customization
effort.
JSON:API is an opinionated specification that removes the need to decide on
many design options commonly included in an API style guide. Representation
format, when to use different HTTP methods, and how to optimize network
connections through response shaping and eager fetching of related resources are
just a few of the decisions already provided by JSON:API.
Listing 7.3 provides an example of a JSON:API message-based representation.
8. http://amundsen.com/hypermedia/hfactor
9. https://jsonapi.org
10. https://github.com/kevinswiber/siren
Selecting a Representation Format 129
{
"data": {
"type": "b ook s",
"id": "12345",
"attrib u tes": {
"isb n": "9 7 8 - 03218 3457 7 ",
"title": "I mplementing D omain- D riv en D esig n",
"description": "W ith I mplementing D omain- D riv en D esig n, V au g h n h as
made an important contrib u tion not only to th e literatu re of th e D omain-
D riv en D esig n commu nity, b u t also to th e literatu re of th e b roader enterprise
application arch itectu re f ield."
},
"relationsh ips": {
"au th ors": {
"data": [
{"id": "7 6 5", "type": "au th ors"}
]
}
},
"inclu ded": [
{
"type": "au th ors",
"id": "7 6 5",
"f u llName": "V au g h n V ernon",
"link s": {
"self ": { "h ref ": "/au th ors/7 6 5" },
"au th oredB ook s": { "h ref ": "/b ook s? au th orI d= 7 6 5" }
}
}
}
}
{
"u b er" :
{
"v ersion" : "1.0",
"data" :
[
{"rel" : [ "self "] , "u rl" : "h ttp://ex ample.org /"},
{"rel" : [ "prof ile"] , "u rl" : "h ttp://ex ample.org /prof iles/b ook s"},
{
"name" : "search B ook s",
"rel" : [ "search ","collection"] ,
"u rl" : "h ttp://ex ample.org /b ook s/search ? q = {q u ery}",
"templated" : "tru e"
},
{
"id" : "b ook - 12345",
"rel" : [ "collection","h ttp://ex ample.org /rels/b ook s"] ,
"u rl" : "h ttp://ex ample.org /b ook s/12345",
"data" : [
{
"name" : "b ook I d",
"v alu e" : "12345",
"lab el" : "B ook I D "
},
{
11. Markus Lanthaler, “Hydra Core Vocabulary: A Vocabulary for Hypermedia-Driven Web APIs”
(Hydra W3C Community Group, 2021), http://www.hydra-cg.com/spec/latest.
12. Mike Amundsen and Irakli Nadareishvili, “Uniform Basis for Exchanging Representations (UBER)”
(2021), https://rawgit.com/uber-hypermedia/specification/master/uber-hypermedia.html.
13. Irakli Nadareishvili and Randall Randall, “Hyper - Foundational Hypermedia Type” (2017), http://
hyperjson.io/spec.html.
14. https://json-ld.org
15. https://www.odata.org
Selecting a Representation Format 131
Notice how the size of the representations grows compared to the more compact
resource serialization formats. With the increased size comes the addition of linked
data and more powerful interactions with API clients. These representation formats
offer more insight into how to navigate related resources and tap into new operations,
including operations that were not available when the client was built.
The goal is to enable generic clients to interact with APIs without the need for custom
code or user interfaces. Instead, a client can interact with an API it has never seen before,
all using the details provided in a semantic, message-based resource representation.
Remember that is always better to include additional details in the message than
to force clients to write more code that infers behavior. This is the essence of why
HTML works so well, as browsers are not required to implement custom code for
every Web site that exists. Instead, the browser implements rendering logic, and the
HTML message is crafted to deliver the desired result. While this may result in a
more verbose message, the result is a more resilient API client that avoids hardcoded
behavior.
Create-Read-Update-Delete
CRUD-based APIs are APIs that offer resource collections that contain instances.
The resources instances will offer some or all of the create, read, update, and delete
lifecycle pattern.
The CRUD pattern may offer a complete or partial CRUD lifecycle around a
resource collection and its instances in a consistent way. The CRUD pattern follows
this familiar pattern:
Using this functional operation approach, article resource instances are able to
support the workflow necessary. In addition, it offers a few advantages:
• Fine-grained access control can be enforced at the API management layer because
each specific action is a unique URL that can be assigned different authoriza-
tion requirements. This avoids coding specific authorization logic into a single
update operation, such as a PU T or PATC H, when the state of a field is changed
• Hypermedia controls are used to signal to clients the possible action(s) availa-
ble based on the user’s authorization scope, as discussed earlier in this chapter.
• The workflow supported by the API is more explicit, as clients don’t have to
look at the PATC H endpoint documentation to understand the valid status values
available, along with re-creating the state machine rules for every API client.
For teams that prefer to avoid this style of functional operations for a resource
instance or collection, an alternative approach is to support hypermedia controls
that reference the same PU T or PATC H operation but support different message struc-
tures based on the type of action being taken.
134 Chapter 7 REST-Based API Design
Singleton Resources
Singleton resources represent a single resource instance outside of a resource
collection. Singleton resources may represent a virtual resource for direct interaction
of an existing resource instance within a collection (e.g., a user’s one and only profile).
APIs may also offer nested singleton resources when there is a one and only one
instance in the relationship between the parent resource and its child resource (e.g., a user’s
configuration). The following examples illustrate possible uses of a singleton resource:
• GET /me—Used in place of GET /u sers/{u serI d}, avoiding the need for consum-
ers to know their own user identifier or risk accessing another user’s data due
to an insecure security configuration
• PU T /u sers/56 7 8 /conf ig u ration—Used to manage a single configuration resource
instance for a specific account
Singleton resources should already exist and therefore should not require a cli-
ent to create them ahead of time. While singleton resources may not offer the full
spectrum of CRUD-style lifecycles like their collection-based brethren, they may still
offer GET, PU T, and/or PATC H HTTP methods.
{
"items": [
{ ... },
{ ... },
{ ... },
{ ... }
]
}
Common REST Design Patterns 135
The server could return the following response to indicate that the request was
valid but that that it could not be processed fully:
HTTP/1.1 202 Accepted
L ocation: h ttps://api.ex ample.com/import- job s/7 9 37
The client could then follow up to determine the status by submitting a request to
the URL provided in the L ocation header:
HTTP/1.1 200 OK
{
"job I d": "7 9 37 ",
"importS tatu s": "I nProg ress",
"percentC omplete": "25",
"su g g estedNex tPollTimestamp": "2018 - 10- 02T11:00:00.00Z ",
"estimatedC ompletionTimestamp": "2018 - 10- 02T14:00:00.00Z "
}
Perhaps a group of four seats is available. Yet, we cannot use the Seats resource
to reserve the seats, as it would require four separate API calls that are not able to be
wrapped in a transaction:
These are only a few of the many design patterns useful for REST and other
API styles. Refer to the API workshop examples16 GitHub repository for
more pattern resources.
Summary
When speaking about REST-based APIs, many conflate the idea of CRUD-based
APIs that use JSON with the REST architectural style. However, REST defines a set
of architectural constraints that help APIs mimic the best aspects of the Web. Of
course, REST APIs may apply various design patterns, including CRUD, to produce
an approachable interaction model for resources.
By applying a five-step design process, a resource-based API design is created that
applies the REST constraints to the API profiles created during the API modeling
process. Mapping the design into machine-readable API description allows tools to
generate documentation for review by the team and the initial set of developers that
will consume the API.
What if REST isn’t the right API style for some or all the APIs identified during
API modeling? Chapter 8 examines how GraphQL and gRPC are two additional API
styles available when REST may not be the best choice or needs to be expanded with
new interaction styles.
16. https://bit.ly/align-define-design-examples
Chapter 8
1. Identify
Digital
Capabilities
7. Document 2. Capture
the API Align Activity
Steps
5. High-Level
4. Model API
Design
Profiles
Figure 8.1 The Design phase offers several options for API styles. Alternatives to REST-based
APIs are detailed in this chapter.
137
138 Chapter 8 RPC and Query-Based API Design
While the REST-based API style comprises most API products available in the market
today, that may not always be the case. Nor is a REST-based API style always the best
option for every API. As an API designer, it is important to understand the options
available and the tradeoffs of each API style to determine the best fit for the target
developers that will consume the API.
Remote procedure call (RPC)–based and query-based API styles are two
additional API styles beyond REST. RPC-based APIs have been available for decades
but have begun to experience a resurgence through the introduction of gRPC. Query-
based APIs are gaining popularity owing to the introduction of GraphQL, making
it the choice for many frontend developers who wish to have greater control over the
shape of API responses.
With multiple API styles available, it is important to understand the advantages
and challenges of each API style. For some API products and platforms, a single API
style may be sufficient. For others, a mixture of intended uses and preferences for the
developers tasked with integrating the API may require a mixture of API styles.
This chapter explores RPC- and query-based API styles and how they may be
used as an alternative or supplement to REST-based (Figure 8.1). The chapter also
defines a design process for RPC- and query-based API styles based on the API
profiles captured during the Define phase outlined in Chapter 6, “API Modeling.”
{"jsonrpc": "2.0", "meth od": "su b tract", "params": [ 42, 23] , "id": 1}
Most RPC-based systems take advantage of a helper library and code generation
tooling to generate the client and server stubs that are responsible for network com-
munications. Those familiar with the fallacies of distributed computing1 recognize
that failures can occur whenever code is executed remotely. While one of RPC’s goals
is to make remote invocation behave as if it is calling a local procedure, network
outages and other failure-handling support is often incorporated into the client and
server stubs and raised as an error.
The remote procedures are defined using an interface definition language (IDL).
Code generators use the IDL to generate the client stub and a server stub skeleton
that is ready for implementation. RPC-based APIs are generally faster to design and
implement for this reason but are less resilient to method renaming and reordering
of parameters.
1. Wikipedia, s.v. “Fallacies of Distributed Computing,” last modified July 24, 2021, 20:52, https://en.
wikipedia.org/wiki/Fallacies_of_distributed_computing.
2. https://developers.google.com/protocol-buffers.
140 Chapter 8 RPC and Query-Based API Design
gRPC Client
est
Requ Stub
gRPC
ffers
c ol Bu on se
Proto Resp
gRPC
gRPC Server Ruby Client
Pro gRP
to c o CR
l Bu esp
ffers ons
e
GoLang Service gRP
CR
equ gRPC Client
est
Stub
Java Client
Figure 8.2 An overview of how gRPC server and client stubs, generated for each programming
language, work together.
By default, gRPC uses the proto file format used by Protocol Buffers to
define each service, the service methods offered, and the messages exchanged.
Listing 8.1 shows an example IDL file for a calculate service that offers a subtract
operation.
• The integration between client and server are tightly coupled. Once in pro-
duction, the order of the fields cannot be changed without breaking API
clients.
• The serialization format for marshaling and unmarshaling of procedure calls
is fixed. Unlike REST-based APIs, multiple media types cannot be used, and
HTTP-based content negotiation is therefore not possible.
• Some RPC protocols, such as gRPC, require custom middleware to work with
browsers and to enforce authorization and role-based access when operations
are tunneled through a single URL.
Finally, keep in mind that gRPC depends on HTTP/2 and requires overriding
default security restrictions to perform considerable customization of HTTP request
headers; browsers cannot support gRPC natively. Instead, projects such as grpc-
web3 offer a library and gateway to transform HTTP/1 requests into gRPC-based
procedure calls.
In summary, RPC-based APIs are best used when the organization owns both the
API client and server. The API team exposes an RPC-based service or API for other
teams within the organization to consume as needed but must strive to keep their
client code up to date with the latest changes.
3. https://github.com/grpc/grpc-web
142 Chapter 8 RPC and Query-Based API Design
Figure 8.3 A table that captures the initial RPC operations based on the previous API profile examples from Chapter 6.
RPC API Design Process
143
144
-categoryId -Book[ ]
-releaseDate or
google.rpc.Status + ProblemDetails
searchBooks( ) Search for books by author, title SearchQuery SearchQueryResponse
-query -Book[ ]
or
google.rpc.Status + ProblemDetails
viewBook( ) View book details ViewBookRequest Book
-bookId or
google.rpc.Status + ProblemDetails
viewCart( ) View the current cart and total ViewCartRequest Cart
-cartId or
google.rpc.Status + ProblemDetails
clearCart( ) Remove all books from the customer’s cart ClearCartRequest Cart
RPC and Query-Based API Design
-cartId or
google.rpc.Status + ProblemDetails
addItemToCart( ) Add a book to the customer’s cart AddCartItemRequest Cart
-cartId or
-quantity google.rpc.Status + ProblemDetails
removeItemFromCart( ) Remove a book from the customer’s cart RemoveCartItemRequest Cart
-cartId or
-cartItemId google.rpc.Status + ProblemDetails
getAuthorDetails( ) Retrieve the details of an author GetAuthorRequest: BookAuthor
-authorId or
google.rpc.Status + ProblemDetails
Figure 8.4 The gRPC design complete with request and response basic message details.
RPC API Design Process 145
Listing 8.2 IDL File for the gRPC Version of the Shopping Cart API
That’s it! The RPC-based API now has a high-level design. Details can now be
added to complete the API and code generators used to jumpstart the development
and integration work. Generating human-readable documentation is also recom-
mended using a tool such as protoc-gen-doc.4
Keep in mind that owing to RPC’s tight coupling with code, many code changes
will have a direct impact on the design of an RPC-based API. Put another way,
RPC-based API designs are replaced, not modified, when code changes are applied.
Notice how most of the effort took place in the API modeling step. By using the
API modeling technique as the foundation of the design effort, the work of bridging
the desired outcomes of the customer is easily mapped into an RPC-based design.
Should additional API styles be required, such as REST, the same API modeling work
can be reapplied to the design effort for the API style of choice.
4. https://github.com/pseudomuto/protoc-gen-doc
What Is a Query-Based API? 147
Understanding OData
Two of the most popular query-based API styles are OData and GraphQL. OData5
is a query-based API protocol that is standardized and managed by OASIS. It is built
upon HTTP and JSON and uses a resource-based approach familiar to those already
familiar with REST.
OData queries are made through specific resource-based URLs via GET. It also
supports hypermedia controls for following related resources and data linking for
expressing resource relationships using hypermedia links rather than identifiers dur-
ing a create or an update operation. OData supports custom actions, which may
mutate data in ways beyond the standard CRUD pattern. Functions are also sup-
ported to support calculations. Listing 8.3 demonstrates the use of a filtered GET to
retrieve any airports located in San Francisco, California, using an OData query.
{
"@ odata.contex t": "/OD ata/$ metadata# Airports",
"v alu e": [
{
"@ odata.id": "/OD ata/Airports( ' KS F O' ) ",
"@ odata.editL ink ": "/OD ata/Airports( ' KS F O' ) ",
"I caoC ode": "KS F O",
"Name": "S an F rancisco I nternational Airport",
5. https://www.odata.org/documentation
148 Chapter 8 RPC and Query-Based API Design
Exploring GraphQL
GraphQL7 is an RPC-based API style that supports the querying and mutation of
data. It is a specification that was developed internally by Facebook in 2012 before
being publicly released in 2015. It was originally designed to overcome the challenges
of supporting Web and mobile clients that need to obtain data via APIs at different
levels of granularity and with the option of retrieving deeply nested graph struc-
tures. Over time, it has become a popular choice by frontend developers who need to
bridge backend data stores with single-page applications (SPAs) and mobile apps.
All GraphQL operations are tunneled through a single HTTP POS T- or GET-based
URL. Requests use the GraphQL query language to shape the response of desired
fields and any nested resources in a single request. Mutations support modifying
data or performing calculation logic and use a similar language to queries to express
the data input for a modification or calculation request. All resource structures are
defined in one or more schema files, ensuring that clients may introspect resources at
design time or runtime. Listing 8.4 provides an example of a GraphQL query.
Listing 8.4 GraphQL Query to Fetch the San Francisco Airport by IATA code
POS T /g raph q l
{
airports( iataC ode : "S F O")
}
{
"data" : {
{
"Name": "S an F rancisco I nternational Airport",
"iataC ode": "S F O",
"L ocation": {
"Address": "S ou th McD onnell R oad, S an F rancisco, C A 9 4128 ",
"C ity": {
"C ou ntryR eg ion": "U nited S tates",
"Name": "S an F rancisco",
"R eg ion": "C alif ornia"
},
7. https://graphql.org
150 Chapter 8 RPC and Query-Based API Design
"L oc": {
"type": "Point",
"coordinates": [
- 122.37 47 22222222,
37 .6 18 8 8 8 8 8 8 8 8 8 9
]
}
}
}
}
}
While GraphQL is popular with frontend developers, it has also gained significant
traction across enterprises as a means to stitch multiple REST APIs together into a
single query-based API. It is also useful for producing query-only reporting APIs
alongside existing REST-based APIs, offering a best-of-breed approach to API
platforms.
Many of the challenges around GraphQL are centered on its choice to tunnel
through a single endpoint rather than take advantage of the full capabilities of HTTP.
This prevents the use of HTTP content negotiation for the support of multiple media
types beyond JSON. It also prevents the use of concurrency controls and optimistic
locking offered by HTTP conditional headers. Similar challenges were experienced
with SOAP-based services, which was designed to work across multiple protocols
including HTTP, SMTP, and JMS-based message brokers.
Enforcing authorization is also a challenge because traditional API gateways that
expect to enforce access control by URL are limited to the single GraphQL operation.
However, some API gateways are extending their capabilities to include authorization
enforcement around GraphQL-based queries and mutations. Likewise, rate limiting,
often associated to a combination of path and HTTP method, must be rethought to
accommodate this new interaction style.
Book Resource
Cart Resource
Property Name Description
cartItems The items currently in
the cart for purchase
subtotal The total cost of all
books in the cart
Cart Item Resource
salesTax The sales tax to be
Property Name Description
applied
book The book currently in
vatTax Any value-added tax
the cart for purchase
to be applied
qty Book Resource
cartTotal The total cost of the
Property Name Description
cart
RPC and Query-Based API Design
Figure 8.6 The Cart resource is the second top-level resource that needs to be supported for the Shopping Cart API modeled in Chapter 6.
Operation Type Operation Name Description Request Response
Figure 8.7 The Shopping Cart API profile, modeled in Chapter 6, is migrated to a tabular format that helps with query-based API design.
Query-Based API Design Process
153
154 Chapter 8 RPC and Query-Based API Design
If the chosen protocol supports only query operations, then mutations must
be handled using a different API style. GraphQL supports both, so the design can
include both query and mutations within the same API definition.
Once the basic operation details have been captured, expand the request and
response columns with further details about the input and output values. These
input and output values were already determined during the API modeling in Chap-
ter 6. Migrate these values into the new API design table. The Shopping Cart API
operations are expanded in Figure 8.8.
type Q u ery {
listB ook s( inpu t: L istB ook sI npu t! ) : B ook sR esponse!
search B ook s( inpu t: S earch B ook sI npu t! ) : B ook sR esponse!
v iew B ook ( inpu t: GetB ook I npu t! ) : B ook S u mmary!
g etC art( inpu t: GetC artI npu t! ) : C art!
g etAu th orD etails( inpu t: GetAu th orD etailsI npu t! ) : B ook Au th or!
}
type Mu tation {
clearC art( ) : C art
addI temToC art( inpu t: AddC artI temI npu t! ) : C art
remov eI temF romC art( inpu t: R emov eC artI temI npu t! ) : C art
}
type B ook sR esponse {
b ook s: [ B ook S u mmary! ]
}
type B ook S u mmary {
b ook I d: S tring !
Query-Based API Design Process 155
isb n: S tring !
title: S tring !
au th ors: [ B ook Au th or! ]
}
type B ook Au th or {
au th orI d: S tring !
f u llName: S tring !
}
type C art {
cartI d: S tring !
cartI tems: [ C artI tem! ]
}
type C artI tem {
cartI temI d: S tring !
b ook I d: S tring !
q u antity: I nt!
}
inpu t L istB ook sI npu t {
of f set: I nt!
limit: I nt!
}
inpu t S earch B ook sI npu t {
q : S tring !
of f set: I nt!
limit: I nt!
}
inpu t GetAu th orD etailsI npu t {
au th orI d: S tring !
}
inpu t AddC artI temI npu t {
cartI d: S tring !
b ook I d: S tring !
q u antity: I nt!
}
inpu t R emov eC artI temI npu t {
cartI d: S tring !
cartI temI d: S tring !
}
156
}
}
Query searchBooks( ) Search for books by author, title query { Book[ ]
Book (searchQuery) {
…
}
}
Query viewBook( ) View book details query { Book
book(bookId) {
...
}
}
Query viewCart( ) View the current cart and total query { Cart
cart(cartId) {
…
RPC and Query-Based API Design
}
}
Mutation clearCart( ) Remove all books from the customer’s mutation clearCart { Cart
cart cartId
}
Mutation addItemToCart( ) Add a book to the customer’s cart mutation addItemToCart { Cart
cartId
bookId
quantity
}
Mutation removeItemFromCart( ) Remove a book from the customer’s mutation removeItemFromCart { Cart
cart cartId
cartItemId
}
Query getAuthorDetails( ) Retrieve the details of an author query { BookAuthor
BookAuthor (authorId) {
…
}
}
Figure 8.8 The Shopping Cart GraphQL API design is now expanded with additional details about queries and mutations.
Summary 157
Summary
REST is not the only API style available. RPC- and query-based APIs provide
additional interaction styles that help developers integrate with an API product or
platform quickly. They may also be combined with REST-based APIs to provide
robust query operations for reporting and fast code generation options.
While the design process is slightly different for each API style, all styles build
upon the investment of aligning the needs of business, customers, and developers.
The next step in the design process is to determine if one or more asynchronous
APIs would benefit the API consumer. This topic is discussed in detail in Chapter 9,
“Messaging, Streaming, and Event-Based Async APIs.”
8. https://www.npmjs.com/package/graphql-docs
9. https://github.com/graphql/graphql-playground
10. https://bit.ly/align-define-design-examples
This page intentionally left blank
Chapter 9
The key to safety lies in the encapsulation. The key to scalability lies in how
messaging is actually done.
— Alan Kay
1. Identify
Digital
Capabilities
7. Document 2. Capture
the API Align Activity
Steps
5. High-Level
4. Model API
Design
Profiles
Figure 9.1 The Design phase offers several options for API styles. This chapter covers asynchronous
API design.
159
160 Chapter 9 Async APIs for Eventing and Streaming
Yet, many developers are forced to build API polling code to constantly check for
changes in server-side state. Building polling code includes additional challenges to
the developer:
• The API sends responses back with default, nonoptimal sorting (e.g., oldest-to-
newest). The consumer must then request all entries to find out if anything new
is available, often keeping a list of the IDs already seen to determine what is new.
• Rate limiting may prevent making requests at the desired intervals to detect
change in a timely fashion.
• The data offered by the API doesn’t provide enough details for the client to
determine if a specific event has occurred, such as a resource modification.
The ideal situation is to have servers inform any interested API consumers about
new data or recent events. However, this isn’t possible with traditional request/
response API styles common with HTTP, as API clients are required to submit a
request before the API server can communicate any changes.
Async APIs help address this need. Rather than constantly polling and
implementing change detection rules on the API client, API servers send asynchronous
push notifications to interested API clients when something on the server has changed.
This opens a whole new series of possibilities compared to traditional, synchronous
Web-based APIs that are rooted in HTTP request/response.
CASE STUDY
GitHub Webhooks Created a New CI/CD Marketplace
GitHub Webhooks have been around for some time, allowing teams to be
notified when new code has been pushed to a GitHub-hosted repository.
While Git supports writing scripts to react to these kinds of events within
a source code repository, GitHub was one of the first vendors to turn these
script-based hooks into Webhooks. Any individual or organization hosting
their code with GitHub could be notified, via an HTTP-based POST, when
new code was available and trigger a new build process.
Over time, continuous integration and delivery (CI/CD) tools that were pre-
viously restricted to on-premises installation could now be offered via a soft-
ware-as-a-service (SaaS) model. These solutions would be granted permission
to receive the Webhook-based notification and start a new build process.
This one async API notification ultimately created an entire SaaS market
of hosted CI/CD tools. That is the power of async APIs.
Figure 9.2 shows an example of each kind of message and the context that it
provides.
Command Message:
Calculate the
Component A 30-day sales Component B
average
Reply Message:
30-day sales
Component A average report Component B
scheduled
Event Message:
30-day sales
Component A average Component B
updated
• Local messaging assumes that messages are sent and received within the
same process. As such, the programming language and host will be the same
as well. The Smalltalk programming language was built to support send-
ing and receiving messages between objects. Actor-based frameworks, such
as Vlingo,1 also support this kind of messaging. A “mailbox” sits between
the code that produces the message and the code that will process the mes-
sage. The consumer code processes each message as soon as possible, some-
times using threads or dedicated CPU cores to process multiple messages in
parallel.
• Interprocess messaging exchanges messages between separate processes but
on the same host. Examples include UNIX sockets and dynamic data exchange
(DDE).
• Distributed messaging involves two or more hosts for messaging. Messages
are transmitted over a network using the desired protocol. Examples of dis-
tributed messaging include message brokers using Advanced Message Queuing
Protocol (AMQP), Message Queuing Telemetry Transport (MQTT), SOAP-
based Web services, REST-based APIs, and so on.
1. https://vlingo.io
A Review of Messaging Fundamentals 165
Protocol
Accept: application/json Semantics
...
Message
<empty>
Message
Body
Protocol
C ontent- Type:application/json Semantics
...
Message
{
"b ook s":[
{"title":"My b ook title",
Message
...}
Body
]
}
Figure 9.3 A REST API example that shows the elements of the request and response messages
exchanged between the API client and the API server.
166 Chapter 9 Async APIs for Eventing and Streaming
2. https://www.rabbitmq.com
3. http://activemq.apache.org/index.html
4. https://github.com/Cicizz/jmqtt
A Review of Messaging Fundamentals 167
Message Message
Component
A A
Message Broker
Component
Message Message
Component
B B
Figure 9.4 A point-to-point queue that dispatches each message to a single message receiver
subscribed to the queue.
168 Chapter 9 Async APIs for Eventing and Streaming
Message
Subscriber A
Message
A
Message
Message Broker
Publisher Subscriber A
Message
Subscriber A
Figure 9.5 A fanout topic that dispatches each message to all subscribed message receivers.
notification of new messages to one or more subscribers, much like message broker
topics. Examples of streaming servers include Apache Kafka,5 Apache Pulsar,6 and
Amazon Kinesis.7
Unlike message brokers, subscribers may request messages at any point from
the topic’s available message history. This allows for the replay messages or for sim-
ply picking up where processing previously left off. Unlike message brokers, most
streaming servers shift state management from the server to the client. The client
is now responsible for tracking the last message seen. Error recovery is also pushed
to the client, forcing the client to resume processing messages at the last known
message.
Support for this style of interaction is accomplished by shifting message manage-
ment from a traditional queue or topic to an append-only log. These logs may store
all messages or limit the history of messages for specified retention period. A topic
using a distributed log with two consumers is shown in Figure 9.6.
Record 10
Record 1
Record 2
Record 3
Record 4
Record 5
Record 6
Record 7
Record 8
Record 9
Topic A
Consumer A Consumer B
Offset: 5 Offset: 10
Figure 9.6 A streaming topic comprising a distributed log of recorded messages, consumed
by two separate consumers who have two separate offsets to reflect their current message.
5. https://kafka.apache.org
6. https://pulsar.apache.org
7. https://aws.amazon.com/kinesis
170 Chapter 9 Async APIs for Eventing and Streaming
With the ability to specify the offset of where they wish to start, clients are able
to solve new kinds of problems using solutions that were not possible with message
brokers:
• Achieve near real-time data processing and data analytics as soon as incoming
data is received from other systems or third parties due to the highly scalable
and low-latency design of message streaming servers.
• Use historical messages to verify the results of code changes prior to pushing
new code to production.
• Execute experimental data analytics against historical messages.
• Remove the need to subscribe to all message broker queues and topics in an
effort to store all messages processed by a message broker for auditing purposes.
• Push data into a data warehouse or data lake for consumption by other sys-
tems, without the need for traditional extract-transform-load (ETL) processes.
The higher scalability of message streaming lends itself to a shift in the way data
is managed and shared. Rather than sharing access to a data store or replicating the
data store, each new or modified data event message is pushed to a topic stream. Any
consumers are then able to process the data change, including storing it locally for
caching or for further analysis.
8. Jeff Lindsay, “Webhooks to Revolutionize the Web” (blog), Wayback Machine, May 3, 2007,
https://web.archive.org/web/20180630220036/http:/progrium.com/blog/2007/05/03/web-hooks-
to-revolutionize-the-web.
9. https://resthooks.org
10. https://resthooks.org/docs
172 Chapter 9 Async APIs for Eventing and Streaming
API Server
Webhook
Dispatcher
Subscriber ...
Subscriber 2
Subscriber 1
Webhook
Webhook
Webhook
Figure 9.7 An API server’s webhook dispatcher that sends a message to each registered URL
that wishes to receive the callback using HTTP POST.
11. https://developer.mozilla.org/en-US/docs/Web/API/EventSource
Async API Styles 173
task _ created:
data: {"id": "6 7 8 9 "}
id: 6 7 8 9
<<Disconnected>>
Figure 9.8 Using Server-Sent Events (SSE) to allow API servers to push events to the client
over a long-lived connection. Connections may be resumed using the Last-Event-Id request
header.
supported media types using content negotiation. Clients interested in using SSE
may do so by specifying the SSE media type instead of JSON or XML in the Accept
request header.
Once connected, the server then pushes new events, separated by a newline. If the
connection is lost for any reason, the client is able to reconnect to start receiving new
events. Clients may provide the L ast- Ev ent- I D HTTP header to recover any missed
events since the last event ID seen by the client. This is useful for failure recovery.
The format for the data field may be any text-based content, from simple data
points to single-line JSON payloads. Multiple lines may be provided using multiple
data-prefixed lines.
SSE supports several use cases:
The W3C SSE specification is easy to read and offers additional specifications and
examples.
12. MDN Web Docs, “Server-Sent Events,” last modified August 10, 2021, https://developer.mozilla.org/
en-US/docs/Web/API/Server-sent_events.
13. Internet Engineering Task Force (IETF), The WebSocket Protocol (Request for Comments 6455,
December 2011), https://tools.ietf.org/html/rfc6455.
14. Internet Assigned Numbers Authority (IANA), WebSocket Protocol Registries, last modified July 19,
2021, https://www.iana.org/assignments/websocket/websocket.xhtml.
API Consumer REST API Server with
Web/Mobile/Server WebSocket support
HTTP/1.1 101 W eb S ock et Protocol Handsh ak e
GET w s://ech o.w eb sock et.org /? encoding = tex t HTTP/1.1 W eb S ock et u pg rade D ate: Th u ,10 S ept 2020 14:53:18 GMT
Orig in: h ttp://w eb sock et.org req u est C onnection: U pg rade
C onnection: U pg rade U pg rade: W eb S ock et
U pg rade: w eb sock et …
W eb S ock et u pg rade
S ec- W eb S ock et- V ersion: 13 response
…
<<WebSocket Enabled>>
Hello, serv er
Hello, client
Figure 9.9 An example interaction between an API client and server using WebSocket and a plain text subprotocol to create a chat application.
Async API Styles
175
176 Chapter 9 Async APIs for Eventing and Streaming
to implement, clients are not able to send requests on the same connection, making
WebSocket a better option when full duplex communication is necessary. Keep this
in mind when choosing an async API style.
gRPC Streaming
TCP protocol is optimized for long-lived bidirectional communications. HTTP/1.1
was built on top of TCP but required multiple connections to allow clients to achieve
concurrency. While this multiconnection requirement is easy to load balance and
therefore scale, it has a considerable performance impact, as each connection requires
establishing a new TCP socket connection and protocol negotiation.
HTTP/2 is a new standard built on the work of the SPDY protocol by Google
to optimize portions of HTTP/1.1. Part of the optimization includes request and
response multiplexing. In response multiplexing, a single HTTP/2 connection is
used for one or more simultaneous requests. This avoids the overhead of creating
new connections for each request, similar to how HTTP/1.1 supports keep-alive
connections. However, HTTP/2 multiplexing allows all requests to be sent at once
rather than sequentially using a keep-alive connection.
In addition, HTTP/2 servers may push resources to the client rather than require
the client to initiate the request. This is a considerable shift from the traditional Web-
based request/response interaction style of HTTP/1.1.
gRPC takes advantage of HTTP/2’s bidirectional communication support,
removing the need to separately support request/response alongside WebSocket,
SSE, or other push-based approaches on top of HTTP/1.1. Because gRPC supports
bidirectional communication, async APIs can be designed and integrated alongside
traditional request/response RPC methods using the same gRPC-based protocol.
There are three options for gRPC streaming: client-to-server, server-to-client, and
bidirectional, illustrated in Figure 9.10.
Like WebSocket, gRPC can send and receive messages and events across a single,
full-duplex connection. Unlike WebSocket, there are no subprotocol decisions to be
made and supported, as gRPC uses Protocol Buffers by default. However, brows-
ers have no built-in gRPC support. The grpc-web project15 is working on bridging
gRPC to browsers, but with limitations. Therefore, gRPC streaming is often limited
to service-to-service interactions.
15. https://github.com/grpc/grpc-web
Async API Styles 177
gRPC Async Option 1: Client Streams to Server gRPC Option 2: Server Streams to Client
Figure 9.10 The three gRPC-based streaming options available: client-to-server, server-to-
client, and bidirectional.
• Webhooks: Webhooks are the only async API style that may be server-
originated—that is, they don’t require the client to initiate a connection first.
Because subscriptions require being able to receive a POS T-based callback, use
Webhooks when server-to-server notification is needed. Web browsers and
mobile apps are unable to take advantage of Webhooks, as they cannot estab-
lish an HTTP server to receive the callback. Subscribers that have inbound
communication restricted by a firewall will not be able to receive the callback,
as there won’t be a network path to the callback server.
• Server-Sent Events: SSE is typically the easiest to implement on the server and
client sides but has limited browser support. It also lacks bidirectional commu-
nication between client and server. Use SSE when there is a need for the server-
push of events that follows RESTful API design.
178 Chapter 9 Async APIs for Eventing and Streaming
Command Messages
Command messages incorporate all of the details necessary to request another com-
ponent to perform a unit of work. When designing commands for async APIs, it is
important to design the command message with sufficient details to process the
request. It may also include a target location where the result message may be pub-
lished. This target location may be a URL to POS T the results, a URI to a message
broker topic, or perhaps a URL to a shared object store such as Amazon S3.
When designing commands, it may be easy to use built-in language mechanisms
such as object serialization to simplify the development of the command producer and
consumer. However, this will limit the systems that will be able to consume and pro-
cess these commands. Instead, seek to use a language-agnostic message format, such
as the UBER hypermedia format, Apache Avro, Protocol Buffers, JSON, or XML.
The following is an example JSON-based command message to request a
customer’s billing address to be updated asynchronously:
{
"messag eType": "cu stomerAddress.u pdate",
"req u estI d": "123f 456 7 ",
"u pdatedAt": "2020- 01- 14T02:56 :45Z ",
Designing Async APIs 179
An additional replyTo field could be provided with the URL for callback, or other
subscribers could listen for a cu stomerAddress.u pdated event to react to the change,
perhaps updating the billing address in a third-party system.
Event Notifications
Event notifications, sometimes referred to as thin events, notify subscribers that a
state change or business event has occurred. They seek to provide only the necessary
information sufficient for the subscriber to determine if the event is of interest.
The event subscriber is responsible for fetching the latest representation of the
details via an API to avoid using stale data. Providing hypermedia links as part
of a thin event helps to integrate API operations for retrieving the latest resource
representation(s) with async APIs such as events. This is shown in the following
example event payload:
{
"ev entType": "cu stomerAddress.u pdated",
"ev entI d": "123e456 7 ",
"u pdatedAt": "2020- 01- 14T03:56 :45Z ",
"cu stomerI d": "330001003",
"_ link s": [
{ "rel": "self ", "h ref ":"/ev ents/123e456 7 " },
{ "rel": "cu stomer", "h ref ":"/cu stomers/330001003" }
]
}
Thin events are used for events related to resources that change frequently, forcing
the event subscriber to retrieve the latest resource representation to avoid working
with stale data. While not necessary, thin events may also include details about
180 Chapter 9 Async APIs for Eventing and Streaming
the specific properties that changed when an update occurred to help consumers
determine if the event is of interest.
• Subscribers want a snapshot of the resource associated with the event rather
than the few details and associated hypermedia links offered by thin events.
• Data state changes are using message streaming to support replaying message
history, requiring a complete point-in-time snapshot of a resource.
• Messaging via async APIs is used for interservice communication, requiring
the publication of a full resource representation to avoid increased API traffic
and tighter coupling between services.
{
"ev entType": "cu stomerAddress.u pdated",
"ev entI d": "123e456 7 ",
"u pdatedAt": "2020- 01- 14T03:56 :45Z ",
"cu stomerI d": "330001003",
"prev iou sB illing AddressL ine1": "...",
"prev iou sB illing AddressL ine2": "...",
"prev iou sB illing AddressC ity": "...",
"prev iou sB illing AddressS tate": "...",
Designing Async APIs 181
{
"ev entType": "cu stomerAddress.u pdated",
"ev entI d": "123e456 7 ",
"u pdatedAt": "2020- 01- 14T03:56 :45Z ",
"cu stomerI d": "330001003",
"prev iou sB illing Address": {
"addressL ine1": "...",
"addressL ine2": "...",
"addressC ity": "...",
"addressS tate": "...",
"addressR eg ionProv ince": "...",
"addressPostalC ode": "..."
},
"new B illing Address": {
"addressL ine1": "...",
"addressL ine2": "...",
"addressC ity": "...",
"addressS tate": "...",
"addressR eg ionProv ince": "...",
"addressPostalC ode": "..."
},
...
}
and easily detect differences in fields or visualize the changes within a user interface
at a later date. Without the pattern, a large value object plus additional coding effort
are required to associate the flattened fields for performing things such as detecting a
difference between the previous and new address.
Event Batching
While most async APIs are designed to notify a subscriber when each message is
available, some designs may benefit from grouping events into a batch. Event
batching requires that subscribers handle one or more messages within each
notification. A simple example is to wrap the notification with an array and enclose
each message within the response, even if there is only one event message at the time:
[
{
"ev entType": "cu stomerAddress.u pdated",
"ev entI d": "123e456 7 ",
"u pdatedAt": "2020- 01- 14T03:56 :45Z ",
"cu stomerI d": "330001003",
"_ link s": [
{ "rel": "self ", "h ref ":"/ev ents/123e456 7 " },
{ "rel": "cu stomer", "h ref ":"/cu stomers/330001003" }
]
},
...,
...
]
Another design option is to provide an envelope that wraps each batch of events
along with additional metadata about the batch:
{
"meta": {
"app- id- 1234",
...
},
"ev ents": [
{
"ev entType": "cu stomerAddress.u pdated",
Designing Async APIs 183
Keep in mind that batching messages or events allows for grouping based on a
specific timeframe, number of events per batch, or through other grouping factors.
Event Ordering
Most event-based systems offer delivery of messages in order when possible.
However, this may not always be the case. Event receivers may go offline and must
restore missing messages while also accepting new inbound messages as they arrive.
Or the message broker is unable to provide the guarantee of ordered message delivery.
In complex distributed systems, multiple brokers and/or message styles may be used
in combination, making it difficult to keep messages in order.
When event ordering is necessary, considerations must be made regarding mes-
sage design. For a single message broker, the broker may offer message sequence
numbering or timestamp-based ordering using the timestamp of when the mes-
sage was received. In distributed architectures, the timestamp cannot be trusted,
as each host may have slight variations in system time, called clock skew. This
requires a centralized sequence-generation technique to be used and assigned to
each message.
Be sure to factor order needs into the message design and across various architec-
tural decisions. It may be necessary to research and understand distributed synchro-
nization using techniques such as a Lamport Clock16 to overcome clock skew across
distributed nodes while ensuring proper ordering of messages across hosts.
16. Wikipedia, s.v. “Lamport Timestamp,” last modified March 22, 2021, 00:201 https://en.wikipedia.
org/wiki/Lamport_timestamp.
184 Chapter 9 Async APIs for Eventing and Streaming
#
# S h opping - API - ev ents- v 1.asyncapi.yaml
#
asyncapi: 2.0.0
inf o:
title: S h opping API Ev ents
v ersion: 1.0.0
description: |
An ex ample of some of th e ev ents pu b lish ed du ring th e b ook store' s sh op-
ping cart ex perience...
ch annels:
b ook s.search ed:
su b scrib e:
messag e:
$ ref : ' # /components/messag es/B ook sS earch ed'
carts.itemAdded:
su b scrib e:
messag e:
$ ref : ' # /components/messag es/C artI temAdded'
components:
messag es:
B ook sS earch ed:
17. https://www.asyncapi.com
Documenting Async APIs 185
payload:
type: ob ject
properties:
q u eryS tring F ilter:
type: string
description: Th e q u ery string u sed in th e search f ilter
categ oryI dF ilter:
type: string
description: Th e categ ory I D u sed in th e search f ilter
releaseD ateF ilter:
type: string
description: Th e release date u sed in th e search f ilter
C artI temAdded:
payload:
type: ob ject
properties:
cartI d:
type: string
description: Th e cartI d w h ere th e b ook w as added
b ook I d:
type: string
description: Th e b ook I D th at w as added to th e cart
q u antity:
type: integ er
description: Th e q u antity of b ook s added
Keep in mind that the AsyncAPI specification also supports the addition of pro-
tocol bindings for each channel’s publish and subscribe messages. This flexibility
allows the same message definition to be used across multiple messaging proto-
cols, including message brokers, SSE, message brokers, and message streams. Visit
the AsyncAPI Web page18 for more information on the specification and additional
resources to help get started using this async API description format. For example
asynchronous API descriptions, refer to the API workshop examples19 available on
GitHub.
18. https://asyncapi.com
19. https://bit.ly/align-define-design-examples
186 Chapter 9 Async APIs for Eventing and Streaming
Summary
Teams can benefit from shifting the API design approach from strictly request/
response APIs to thinking in terms of how APIs can offer both synchronous request/
response operations and asynchronous events. These events enable the API to push
notifications to other teams that can build entirely new capabilities and perhaps
product offerings on top of the original API. The result will be increased innovation
and more transformative APIs as part of an API product or API platform initiative.
Part V
187
This page intentionally left blank
Chapter 10
From APIs
to Microservices
The biggest fallacy about monoliths is you can have only one.
— Kelsey Hightower
1. Identify
Digital
Capabilities
7. Document 2. Capture
the API Align Activity
Steps
5. High-Level
4. Model API
Design
Profiles
Figure 10.1 Refining the API design may include decomposing it into smaller services to
reduce the overall complexity of the solution.
189
190 Chapter 10 From APIs to Microservices
All organizations want to deliver business value as fast as possible. At the same time,
they must ensure their software consistently works as expected. Increasing the speed
of development risks an increase in bugs and decrease in reliability. The larger a soft-
ware solution becomes, the greater this risk.
To mitigate these risks, organizations are required to reduce velocity in software
delivery by coordinating through meetings. These meetings seek to optimize delivery
while addressing any risks along the way. The larger the software solution, the more
meetings that are required to mitigate associated risks. Yet, every meeting slows
down the delivery process. Therefore, the balance between speed and delivering
quality software is important.
Decomposing APIs into microservices (Figure 10.1) is one option for teams
to address this need for balance. This chapter explores the topic of microservices,
including benefits, challenges, and alternatives to microservices.
Figure 10.2 The traditional way of thinking about monolithic, service-oriented, and
microservice architectures. Dashed lines represent traditional course-grained boundaries that
are further decomposed to reduce the complexity of more course-grained services.
What Are Microservices? 191
User Interface
DB DB DB
Figure 10.3 Microservices decompose high complexity into smaller, independently deployable
components.
Testing becomes more approachable, and automated test suites become more
focused on the single component (see Figure 10.3).
The idea of microservices has been around for more than a decade but only
recently have microservices gained widespread use by the late majority. In the early
days of microservices, teams had to weigh the effort required to establish and main-
tain the infrastructure necessary for a microservices architecture. Over time, many
of these factors were addressed through cloud-native infrastructure, the growth of
the DevOps culture, better delivery pipeline automation, and the use of containeri-
zation for producing self-contained deployment packages.
1. Wikipedia, s.v. “Metcalfe’s Law,” last modified April 13, 2021, 13:48, https://en.wikipedia.org/wiki/
Metcalfe%27s_law.
Weighing the Complexity of Microservices 193
2. Vaughn Vernon and Tomasz Jaskula, Strategic Monoliths and Microservices: Driving Innovation
Using Purposeful Architecture (Boston: Addison-Wesley, 2021).
194 Chapter 10 From APIs to Microservices
spread across components, making each individual component easier to build and
manage. However, separating the problem into distributed components introduces
other complexity.
Each team and organization must consider both the complexity of a solution
and the complexities that microservices introduce to determine if a shift to
microservices will help or hinder the organization’s ability to deliver solutions with
both speed and safety. While a single microservice may offer lower complexity, the
infrastructure and automation requirements to deliver, monitor, and protect the
service at runtime increases.
If the solution has a low factor of complexity, then microservices are often
unnecessary and may even be detrimental to solution success. If the complexity of
the solution is unknown, weigh the factors that follow and then consider starting
with a minimal solution that balances these factors, migrating to microservices when
and if the complexity increases.
Self-Service Infrastructure
Microservices require a self-service, fully automated infrastructure. Teams must be
able to design a microservice, build it, and deploy code without any manual processes
or approvals. Organizations that have not fully automated their provisioning and
deployment pipeline will encounter considerable friction. Without full automation
support, new code will be added to existing microservices to avoid manual processes,
resulting in a few very large, siloed services.
Tip
Don’t discount the organizational and cultural impacts of adopting micros-
ervices. The shift from product- or project-based ownership to the ownership
of one or a few microservices within a bounded area will have an impact on
reporting structures and team alignment. Count the cost before proceeding.
Otherwise, the organization may be trading code complexity for organizational
complexity.
3. Wikipedia, s.v. “Fallacies of Distributed Computing,” last modified July 24, 2021, 20:52, https://
en.wikipedia.org/wiki/Fallacies_of_distributed_computing.
Weighing the Complexity of Microservices 197
4. Chris Richardson, “Pattern: Saga,” Microservice Architecture, accessed August 19, 2021, https://
microservices.io/patterns/data/saga.html.
198 Chapter 10 From APIs to Microservices
X
Service A Service B X Service C
serviceC.calcMoreStuff()
Figure 10.4 Microservices that use a synchronous, request/response style result in call chains that can fail without the client’s knowledge.
Synchronous and Asynchronous Microservices
199
200
notification of result
serviceA.calcSomething( )
Chapter 10
Service A
Service Client
Queue
Service B
)
Service B
lse( Queue
h ingE
So met
calc
ic eB.
serv Service B
Service A Service A se
notification of result rvi
ce
C.
ca
lcM
or
eS
tuff
()
From APIs to Microservices
Service C
Service C
Queue
Service C
notification of result
Figure 10.5 Asynchronous microservices are able to receive command messages and respond with results without the need for fragile call
chaining.
Microservice Architecture Styles 201
API-Based Orchestration
This style starts with the design of an API that is further decomposed into
microservices as appropriate. The API becomes the stable orchestration layer
across one or more microservices, offering a more stable contract externally while
202 Chapter 10 From APIs to Microservices
Figure 10.6 Direct service communication allows any service to invoke any other service.
API
API
API
Gateway
API
Figure 10.7 The API-based orchestration style offers increased contract stability while
hiding the internal microservices.
Microservice Architecture Styles 203
Cell-Based Architecture
A cell-based architecture (see Figure 10.8) blends the previous two styles to bring a
more modular approach to microservices. Each cell offers one or more digital capa-
bilities, provided through a synchronous or asynchronous API. The API is external-
ized via a gateway, hiding the internal details of service decomposition through
encapsulation. Cells are combined to create larger solutions. Because of the modular
composability of this style, it is often found in large organizations, as it offers better
management for their evolving systems.
Uber Engineering recently shifted from the integration of many small services
to this cell-based architecture model. It discovered that complexity increases are far
outweighed by the value that microservices provided. Uber refers to this approach as
Domain-Oriented Microservice Architecture (DOMA) and offers a helpful article5
that summarizes the approach. It resembles many of the elements of a cell-based
architecture by reducing the complexity of a large-scale microservice architecture
while maintaining the flexibility and benefits that it provides.
API
API
Gateway
API
Gateway
API
Gateway
API
Figure 10.8 A cell-based architecture blends the direct service communication and API-
based orchestration styles into a more modular approach for large organizations or complex
systems.
5. Adam Gluck, “Introducing Domain-Oriented Microservice Architecture,” Uber Engineering, July 23,
2020, https://eng.uber.com/microservice-architecture.
204 Chapter 10 From APIs to Microservices
Right-Sizing Microservices
Organizations on the path to microservices often struggle with finding the right size
for microservices. Teams often ask, “What is the maximum allowable size for a
microservice?” A better question would be, “What is the right size for this microservice
based on what is needed today?”
Microservices aren’t frozen in time. Instead, they grow and become more complex.
Over time, a microservice may need to be split. At other times, two microservices
may become codependent and benefit from being combined into a single service.
Therefore, the size of a microservice will change over time.
It is also important to note that services tend to grow over time, requiring that the
boundaries of a microservice be reevaluated often. This can only be done efficiently
if service ownership resides with a single team. Services shared across teams require
further coordination meetings.
Right-sizing microservices requires a continuous process of design and reevaluation:
Tip
It is best to focus less on the size of the microservice and more on the purpose of
the service. Microservices should seek to make future change easier, even if that
means the service is more course-grained at the start.
Shopping Search
Customer
API Engine
GET /b ook s
B ook s[ ]
C art{cartI d: 456 }
D EL ETE/carts/456 /items/1
B ook s[ ]
GET /carts/456
Shopping Search
Customer
API Engine
Figure 10.9 Expanded Shopping API web sequence diagram that now includes any
external system.
206 Chapter 10 From APIs to Microservices
Shopping Search
Customer
API Engine
GET /b ook s
B ook s[ ]
C art{cartI d: 456 }
B ook s[ ]
GET /carts/456
Shopping Search
Customer
API Engine
Figure 10.10 The search engine integration will require specialized knowledge of how to
properly index and search entities, so the Search Books operation is a good candidate for a
separate microservice.
ensuring the search engine indexes are both performant and deliver the search capa-
bilities required by customers. Figure 10.10 show the boundary for the candidate
microservice that will support book searches.
GET /b ook s
B ook [ ]
C art{cartI d: 456 }
B ook [ ]
GET /carts/456
Figure 10.11 Updated sequence diagram with the candidate microservice involved, which
Customer
API Service Engine
Decomposing APIs into Microservices
207
208 Chapter 10 From APIs to Microservices
Review the updates and determine if the candidate microservice is doing too
much and should be further decomposed. Or, perhaps the service is doing too little,
introducing too many network calls and therefore should be combined into a slightly
larger service.
6. https://launchany.com/canvas
Third-party search engine, e.g., Elasticsearch searchBooks(query)
Decomposing APIs into Microservices
Figure 10.12 A Microservice Design Canvas that captures the candidate microservice, including design considerations, prior to implementation.
209
210 Chapter 10 From APIs to Microservices
In addition, consider whether a dedicated team will own the microservice. If so,
does the introduction of the candidate microservice reduce or increase cross-team
coordination? Not all decisions about service decomposition are about reducing
code size.
Finally, avoid splitting services based on the CRUD lifecycle, creating one service
per operation (e.g., Create Project Service, Update Project Service, Read Project
Service, List Projects Service, Delete Project Service). This pattern creates more
coordination requirements between each service team. A change to the resource
representation for a project requires coordinating with each of the teams that own
the service. The exception is when complexity dictates the need to split one part of a
CRUD lifecycle due to increased complexity. For example, the complexity of payment
processing integration may merit shifting this behavior to a separate microservice.
Summary
Microservices are independently deployable units of code that are combined to
create distributed systems. Moving to microservices requires a combination of new
technologies and top-down organizational support. After an organization reflects
carefully on the decision factors, its shift to microservices may result in the primary
benefit of reduced coordination costs across multiple teams.
Be wary of technology trends that do not inject more benefit than the complexity
they require. Microservices have offered benefits to some organizations, but
not without their challenges. Organizations must count the cost of moving to
microservices to determine if the complexity of designing, building, and operating
microservices outweighs the complexity of a single, monolithic codebase.
Alternatives, such as modular monoliths and cell-based architectures, support
many of the goals of microservices but with varying support for reduced coordination
and local decision optimization. When in doubt, apply the “you ain’t gonna need it”
(YAGNI) principle of agile software by starting with a modular monolith API and
decomposing it into microservices when the need arises.
This page intentionally left blank
Chapter 11
Improving the
Developer Experience
Every useful API that delivers value will typically have multiple consumers.
This is a natural asymmetry, which will only increase over time.
— Mark O’Neill
1. Identify
Digital
Capabilities
7. Document 2. Capture
the API Align Activity
Steps
5. High-Level
4. Model API
Design
Profiles
Figure 11.1 Refining the API design includes improving the developer experience through
helper libraries and command-line interfaces.
213
214 Chapter 11 Improving the Developer Experience
When teams think about delivering an API, the primary focus is on the code that
must be built. They focus on considerations such as the target programming lan-
guage, frameworks that aid in building the API, continuous integration and delivery
(CI/CD) pipelines, and other factors. While all of these decisions are important, they
involve the API provider only. They do not directly empower the tens, hundreds, or
thousands of future API consumers that will use the API.
As an API provider, it is important to keep the API consumers first in everything
that is designed and delivered. This responsibility includes creating mock APIs
to help early adopters provide feedback early on the API design (Figure 11.1). It
also requires consideration of whether to offer helper libraries and command-
line interfaces to reduce the integration time by developers across all skill levels
consuming the API. This chapter addresses these concerns in an effort to multiply
the impact across the many current and future API consumers that will integrate
the API.
be used to drive automated test creation. Over time, the mock integration is replaced
with the actual API until the mock is no longer needed and is removed completely.
The interface remains constant, but the implementation is replaced over time.
Meanwhile, teams are able to proceed in parallel.
There are three primary types of API mock implementations: static mocking,
prototyping, and README-based mocking. Each may be used independently or
in combination to explore an API design prior to delivery. Mocks may also be used
for standing up a local or cloud-based learning environment that is separate from
production.
{
"data": {
"type": "b ook s",
"id": "12345",
"attrib u tes": {
"isb n": "9 7 8 - 03218 3457 7 ",
"title": "I mplementing D omain- D riv en D esig n",
"description": "W ith I mplementing D omain- D riv en D esig n, V au g h n h as
made an important contrib u tion not only to th e literatu re of th e D omain-
D riv en D esig n commu nity, b u t also to th e literatu re of th e b roader enterprise
application arch itectu re f ield."
},
"relationsh ips": {
"au th ors": {
"data": [
{"id": "7 6 5", "type": "au th ors"}
]
}
},
216 Chapter 11 Improving the Developer Experience
"inclu ded": [
{
"type": "au th ors",
"id": "7 6 5",
"f u llName": "V au g h n V ernon",
"link s": {
"self ": { "h ref ": "/au th ors/7 6 5" },
"au th oredB ook s": { "h ref ": "/b ook s? au th orI d= 7 6 5" }
}
}
}
}
Static mocks may be provided using a Web server, such as Apache or nginx, to
allow frontend developers to integrate the mock API responses into the user interface.
They will then be able to provide feedback early and often as they start to parse and
integrate the static mocks into their code.
It is important to note that static mocks lack any implementation, so mock inte-
gration will be limited to GET-based operations only. However, creating a static mock
of an API operation that retrieves a resource representation is quite useful, easy to
build, and provides opportunities for plenty of feedback.
Note
Teams may wish to select a language and framework that isn’t supported for
production by the organization. Doing so will ensure that prototypes intended to
be throwaway don’t suddenly become production code.
Creating a Mock API Implementation 217
The use of an API mocking tool, often based on an API description format such
as the OpenAPI Specification (OAS), allows teams to skip most or all development
efforts. These tools produce simple mock implementations that store data
temporarily for common create-read-update-delete (CRUD)–based operations.
Some tools generate code for the mock implementation, and others create the mock
API on the fly.
It is recommended to keep API prototypes simple at first. Expand the prototype
as needed to deep-dive into any contentious areas that need further exploration or
areas that can encourage parallel development.
README-Based Mocking
README-based mocking provides an alternative prototyping style without the need
to write code. A README file is created to demonstrate how to use an API to
accomplish one or more desired outcomes. README-based mocks help to validate
the API design before implementation starts by sharing the intent of API usage to
produce desired outcomes.
Most README-based mocks use Markdown, enabling the combination of text
and code examples to be easily produced and rendered in a browser. Tools such as
GitHub and GitLab have built-in Markdown support, although static site generation
tools such as Jekyll or Hugo may also be used.
Following is a README-based mock that demonstrates how to retrieve book
details, then add the book to a cart using the JSON:API media format:
HTTP/1.1 200 OK
C ontent- Type: application/v nd.api+ json
...
{
"data": {
"type": "b ook s",
"id": "12345",
"attrib u tes": {
"isb n": "9 7 8 - 03218 3457 7 ",
218 Chapter 11 Improving the Developer Experience
{
"data": {
"type": "carts",
"id": "6 7 8 9 ",
"attrib u tes": {
Providing Helper Libraries and SDKs 219
...
Using this approach gives teams time to think through the API design and how
it will be used to produce outcomes—without the overhead of writing or changing
code. It also increases the quality of documentation and the surrounding conver-
sations about the design. README-driven design can be thought of as the hand-
written version of an acceptance test using behavior-driven development (BDD)
frameworks such as Cucumber.1
1. https://cucumber.io
220 Chapter 11 Improving the Developer Experience
API teams must determine how they plan to provide helper libraries, which
programming languages they plan to support, and how community- or consumer-
generated helper libraries may impact their developer support program.
Including SDK language and version number in the U ser- Ag ent header for all
requests can help. However, the most important factor is to ensure that everything is
logged on the client side and server side.
Support emails will become more confusing when trying to determine the
language, helper library version, and API version being used. Add community-
contributed helper libraries into the process, and more confusion will occur. This
confusion can exist for even the most experienced developer.
The addition of a request identifier or correlation identifier is a common solution
to this problem. These identifiers help to correlate client requests with server-side
logs as developers correspond with API support team. Application performance
management (APM) tools may be useful for diagnosing issues as well.
Offering a CLI tool expands the reach of an API beyond full-time developers
to automation engineers who are better equipped to write shell scripts rather than
applications to integrate with APIs. CLI tools may offer human-friendly output, in
addition to JSON, CSV, or other output formats that support better automation and
tool chaining.
Designing a CLI tool that wraps an API is no different than designing the API
itself. It requires understanding the desired outcomes, activities, and steps required to
accomplish the jobs to be done (JTBD). Then, design the CLI interface to meet these
outcomes. The following code block shows how a CLI interface could be designed to
support the Shopping API designed in previous chapters:
S u ccess!
C art S u mmary:
C art I tems:
$ > ...
Offering CLIs for APIs 223
To offer a great CLI experience, API teams need to become students of human-
first CLI design. The excellent Command Line Interface Guidelines2 site offers in-
depth details on how to design a human-first CLI based on 40 years of patterns and
practices across tooling and operating systems.
Also, teams should seek to understand the pipe and filter design pattern commonly
seen across *nix tools such as sed, aw k , and g rep to better understand how tool chain-
ing works. Finally, carefully examining popular CLIs from Kubernetes, Heroku, and
others help teams to see how to design a user-friendly CLI that wraps remote APIs.
Whether a small team is tasked with delivering multiple APIs in quick succes-
sion or an organization is scaling its API program, leveraging code-generation
tools is essential. Code generation ensures APIs are delivered consistently and
at scale by incorporating boilerplate code and common patterns. While some
API styles such as gRPC rely heavily on code generation, other API styles
consider code-generation support as optional. Code generators are helpful
to generate SDKs and helper libraries consistently across a variety of target
programming languages.
For REST-based APIs, the Swagger Codegen3 project is the most popu-
lar. This project offers open-source client-side code generators for a variety
of programming languages. Another popular option for REST-based APIs
is APIMatic,4 which is a freemium tool that offers code-generation support.
All of these tools generate client code based on an OAS description file. The
resulting code may be packaged up and distributed by the API team.
Some organizations have found that creating their own client-side code
generators is a better option. While doing so requires more investment, the
generated code may be customized as needed. For example, code can be
customized to track rate limiting, detect special error response codes, and
incorporate retry loops where appropriate.
2. Aanand Prasad, Ben Firshman, Carl Tashian, and Eva Parish, Command Line Interface Guidelines,
accessed August 20, 2021, https://clig.dev.
3. https://swagger.io/tools/swagger-codegen
4. https://www.apimatic.io
224 Chapter 11 Improving the Developer Experience
Summary
API design doesn’t stop with the details of API operations and protocol semantics. It
requires thoughtful consideration regarding how the API will be integrated by
developers. While some code decisions are important for the API provider, these are
internal concerns that do not have a direct impact on the many API consumers that
will use the API. The more complex the API, the more tooling (e.g., API mocks,
helper libraries, CLIs) is required to support the design and delivery process.
API teams must consider how their decisions may have a positive or negative
impact on future API consumers. Avoid making decisions that provide local
optimizations for a few developers, instead opting to make global optimizations for
the many current and future consumers of the API.
Chapter 12
1. Identify
Digital
Capabilities
7. Document 2. Capture
the API Align Activity
Steps
5. High-Level
4. Model API
Design
Profiles
Figure 12.1 API testing refines the API design by identifying API quality issues early.
225
226 Chapter 12 API Testing Strategies
Acceptance Testing
Acceptance testing, also called solution-oriented testing, ensures that the API
supports the captured job stories. It seeks to answer the following questions:
• Does the API solve real problems that our customers have?
• Does it produce the desired outcomes for the jobs to be done?
While not typically associated with automated testing, security testing is an active
process that includes design-time review processes, development-time static and
dynamic code analysis, and runtime monitoring.
API Contract Testing 227
Operational Monitoring
APIs can and often do provide the primary interface for applications to interact
with a system. Because the API service plays the role of a dependency, it is critical
for the service to be available, whether to other services that are internal to an
organization or to external partners and customers. In addition, there may be
service-level agreements (SLAs) that the company has agreed to undertake with
customers and partners regarding the performance and uptime of an API. Failing
to meet an SLA could yield a negative financial result as well as angry or upset
customers.
Operational monitoring answers the following questions:
Some team members may suggest that building dedicated tests for an API is
wasteful. They may attempt to make the case that user interface (UI) tests
cover the API sufficiently, given that the UI calls the API. However, this is not
Selecting Tools to Accelerate Testing 229
the case. Instead, the UI tests the API only as far as the UI exercises the API.
This means that if the UI is performing client-side validation of user input,
then UI tests would never verify the API’s ability to handle bad data.
While some may say that this level of testing is sufficient, they may be for-
getting the recommendation of the Open Web Application Security Project
(OWASP): do not trust user input. A user or client will not always submit
data in a way that an API expects. Always validate the data that comes from
forms as well as from HTTP request headers.
One of the goals of API testing is to ensure that the API is able to handle a
multitude of good and bad values that may be submitted outside of a specific
UI. If we depend only on UI tests, then the API should not be considered suf-
ficiently tested.
Another goal of API testing is to ensure that the API cannot be deployed
into production without passing tests. This requires that API tests become
part of the continuous integration and delivery pipeline, just like all other
types of automated testing.
API testing is often automated through code or test scripts and executed in a
dedicated test environment. Automating these tests has a higher infrastructure
cost because of the need for additional nonproduction environments that contain
infrastructure resources. Be sure to take into consideration how tests will be
automated and the infrastructure cost required to support them.
Finally, consider how test-driven development (TDD) may be extended through
the strategic selection of API testing tools. Dedicated QA teams may build automated
test suites that can be executed by developers as they implement the API. Developers
who are tasked with writing the API tests themselves may wish to take a similar
approach, much like they apply TDD to their day-to-day development process. This
approach helps to demonstrate progress and validate that an API implementation
handles all success, invalid, and error cases.
1. https://mockaroo.com
Summary 231
Summary
A robust API testing strategy is an important step to API delivery and is a formidable
foe against regressions sneaking into an API. A proper API testing strategy helps to
ensure API correctness and reliability while ensuring the desired outcomes are
achievable. It should also extend beyond the development phase and into runtime
testing to maintain a secure and performance environment. An API should not be
considered complete until all tests have been created, executed, and passed.
This page intentionally left blank
Chapter 13
Documentation is the third user interface for APIs, and the most important.
— D. Keith Casey
1. Identify
Digital
Capabilities
7. Document 2. Capture
the API Align Activity
Steps
5. High-Level
4. Model API
Design
Profiles
Figure 13.1 The final step in the Refine phase is to produce robust documentation,
incorporating learnings back into the API design.
233
234 Chapter 13 Document the API Design
OpenAPI Specification
Formerly known as Swagger, the OpenAPI Specification (OAS) is currently one of
the most popular formats for describing the details for an API. It is managed by the
Linux Foundation under the stewardship of the OpenAPI Initiative (OAI). The
Swagger brand is now owned by SmartBear, which continues to maintain and
support various open-source API projects under the Swagger name.
OAS came into popularity owing to the try-it-out feature that is built into the
SwaggerUI project. This project was designed to generate HTML-based API
reference documentation for developers. The try-it-out feature allows developers
and nondevelopers to explore an API against a live server from within the generated
documentation. It supports JSON and YAML-based formats.
OAS is currently in version 3 of the specification, although OAS v2 is still
encountered in some organizations and open-source projects. The tools ecosystem is
vast and continues to grow, making this a popular choice for teams building their first
or thirty-first API. An example of OAS v3 is provided in Listing 13.1, based on the API
design for the Shopping Cart API created in Chapter 7, “REST-Based API Design.”
openapi: 3.0.0
inf o:
title: B ook store S h opping Ex ample
description: Th e B ook store Ex ample R ES T- b ased API su pports th e sh opping
1. https://bit.ly/align-define-design-examples
236 Chapter 13 Document the API Design
b ook s:
type: array
items:
$ ref : ' # /components/sch emas/B ook S u mmary'
description: "A list of b ook su mmaries as a resu lt of a list or
f ilter req u est..."
B ook S u mmary:
title: B ook S u mmary
type: ob ject
properties:
b ook I d:
type: string
description: An internal identif ier, separate f rom th e I S B N, th at
identif ies th e b ook w ith in th e inv entory
isb n:
type: string
description: Th e I S B N of th e b ook
title:
type: string
description: "Th e b ook title, e.g ., A Practical Approach to API
D esig n"
au th ors:
type: array
items:
$ ref : ' # /components/sch emas/B ook Au th or'
description: ' '
description: "S u mmariz es a b ook th at is stock ed b y th e b ook store..."
B ook Au th or:
title: B ook Au th or
type: ob ject
properties:
au th orI d:
type: string
description: An internal identif ier th at ref erences th e au th or
f u llName:
type: string
description: "Th e f u ll name of th e au th or, e.g ., D . Keith C asey"
description: "R epresents a sing le au th or f or a b ook . S ince a b ook may
h av e more th an one au th or, ..."
238 Chapter 13 Document the API Design
API Blueprint
API Blueprint originated from an API tools vendor called Apiary, now a part of
Oracle. It combines the idea of easy documentation generation using Markdown
with a structure that makes it machine-readable for supporting code generation and
other tooling needs.
Because API Blueprint is based on Markdown, any tool capable of rendering
and editing files using the Markdown format, including integrated development
environments (IDEs), is able to work with this format. While the ecosystem of
tooling isn’t as vast as that of OAS, it does have considerable community support
owing to the preacquisition efforts of Apiary. As Listing 13.2 shows, it is easy to
work with and therefore a popular choice for those seeking to combine Markdown-
based documentation with a machine-readable API description format.
F OR MAT: 1A
HOS T: h ttps://w w w .ex ample.com
# Grou p B ook s
+ Parameters
+ q ( string , optional)
A q u ery string to u se f or f iltering b ook s b y title and description.
I f not prov ided, all av ailab le b ook s w ill b e listed...
+ of f set ( nu mb er, optional) -
A of f set f rom w h ich th e list of b ook s are retriev ed, w h ere an of f set
of 0 means th e f irst pag e of resu lts...
+ D ef au lt: 0
API Description Formats 239
# # # Properties
# # # Properties
+ ' b ook I d' ( string , optional) - An internal identif ier, separate f rom th e
I S B N, th at identif ies th e b ook w ith in th e inv entory
+ ' isb n' ( string , optional) - Th e I S B N of th e b ook
+ ' title' ( string , optional) - Th e b ook title, e.g ., A Practical Approach
to API D esig n
+ ' au th ors' ( array[ B ook Au th or] , optional)
# # B ook Au th or ( ob ject)
R epresents a sing le au th or f or a b ook . S ince a b ook may h av e more th an one
au th or, ...
240 Chapter 13 Document the API Design
# # # Properties
+ ' au th orI d' ( string , optional) - An internal identif ier th at ref erences th e
au th or
+ ' f u llName' ( string , optional) - Th e f u ll name of th e au th or, e.g ., D . Keith
C asey
RAML
RAML stands for RESTful API Modeling Language and was designed with the full
API design lifecycle in mind. It originated within MuleSoft but included contributors
from many other industry leaders. The design of RAML was intended to support
design tooling alongside documentation and code-generation tools. RAML is built
on the YAML format.
While RAML originated with the help of MuleSoft, the specification and much
of the tooling is vendor neutral. RAML focuses on describing resources, methods,
parameters, responses, media types, and other HTTP constructs common to REST-
based APIs. However, it can be used to describe nearly any HTTP-based API format.
Listing 13.3 uses RAML for the Shopping Cart API.
# % R AML 1.0
title: B ook store S h opping API Ex ample
v ersion: 1.0
b aseU ri: h ttps://w w w .ex ample.com
b aseU riParameters:
def au ltHost:
req u ired: f alse
def au lt: w w w .ex ample.com
ex ample:
v alu e: w w w .ex ample.com
displayName: def au ltHost
type: string
protocols:
- HTTPS
docu mentation:
- title: B ook store S h opping API Ex ample
content: Th e B ook store Ex ample R ES T- b ased API su pports th e sh opping ex pe-
rience of an online b ook store. Th e API inclu des th e f ollow ing capab ilities
and operations...
API Description Formats 241
types:
L istB ook sR esponse:
displayName: L istB ook sR esponse
description: A list of b ook su mmaries as a resu lt of a list or f ilter
req u est...
type: ob ject
properties:
b ook s:
req u ired: f alse
displayName: b ook s
type: array
items:
type: B ook S u mmary
B ook S u mmary:
displayName: B ook S u mmary
description: S u mmariz es a b ook th at is stock ed b y th e b ook store...
type: ob ject
properties:
b ook I d:
req u ired: f alse
displayName: b ook I d
description: An internal identif ier, separate f rom th e I S B N, th at
identif ies th e b ook w ith in th e inv entory
type: string
isb n:
req u ired: f alse
displayName: isb n
description: Th e I S B N of th e b ook
type: string
title:
req u ired: f alse
displayName: title
description: Th e b ook title, e.g ., A Practical Approach to API
D esig n
type: string
au th ors:
req u ired: f alse
displayName: au th ors
type: array
242 Chapter 13 Document the API Design
items:
type: B ook Au th or
B ook Au th or:
displayName: B ook Au th or
description: R epresents a sing le au th or f or a b ook . S ince a b ook may
h av e more th an one au th or, ...
type: ob ject
properties:
au th orI d:
req u ired: f alse
displayName: au th orI d
description: An internal identif ier th at ref erences th e au th or
type: string
f u llName:
req u ired: f alse
displayName: f u llName
description: Th e f u ll name of th e au th or, e.g ., D . Keith C asey
type: string
/b ook s:
g et:
displayName: L istB ook s
description: Prov ides a pag inated list of b ook s b ased on th e search
criteria prov ided...
q u eryParameters:
q :
req u ired: f alse
displayName: q
description: A q u ery string to u se f or f iltering b ook s b y title and
description. I f not prov ided, all av ailab le b ook s w ill b e listed...
type: string
of f set:
req u ired: f alse
def au lt: 0
ex ample:
v alu e: 0
displayName: of f set
description: A of f set f rom w h ich th e list of b ook s are retriev ed,
w h ere an of f set of 0 means th e f irst pag e of resu lts...
type: integ er
minimu m: 0
f ormat: int32
API Description Formats 243
limit:
req u ired: f alse
def au lt: 25
ex ample:
v alu e: 25
displayName: limit
description: Nu mb er of records to b e inclu ded in API call, def au lting
to 25 records at a time if not prov ided...
type: integ er
minimu m: 1
max imu m: 100
f ormat: int32
h eaders:
Au th oriz ation:
req u ired: tru e
displayName: Au th oriz ation
description: An OAu th 2.0 access tok en th at au th oriz es you r app
to call th is operation...
type: string
responses:
200:
description: S u ccess
h eaders:
C ontent- Type:
def au lt: application/json
displayName: C ontent- Type
type: string
b ody:
application/json:
displayName: response
description: S u ccess
type: L istB ook sR esponse
401:
description: R eq u est f ailed. R eceiv ed w h en a req u est is made w ith
inv alid API credentials...
b ody: {}
403:
description: R eq u est f ailed. R eceiv ed w h en a req u est is made w ith
v alid API credentials tow ards an API operation or resou rce you do not h av e
access to.
b ody: {}
244 Chapter 13 Document the API Design
JSON Schema
The JSON Schema specification offers a machine-readable format for capturing the
structure and validation rules for JSON-based structures. The specification is divided
into core foundational rules and validation rules, making it a comprehensive solution
for defining JSON schemas that require validation. JSON Schema can be thought of
as the JSON equivalent to XML Schema.
While independent of any one API style, JSON Schema may be used to describe
resource representations for REST-based APIs and other API styles. It is also found
in organizations as a single format for defining schema formats for domain objects
across the enterprise.
While the schema definition portion of OAS is quite flexible, it lacks some of the
robust definition support offered by JSON Schema. Recent efforts with OAS v3.1
have helped to bring JSON Schema and OAS into alignment to allow for the use
of both formats. Expect JSON Schema to continue gaining tooling support moving
forward given its acceptance within the OAS description format. JSON Schema is
demonstrated in Listing 13.4.
{
"$ id": "h ttps://ex ample.com/B ook S u mmary.sch ema.json",
"$ sch ema": "h ttp://json- sch ema.org /draf t- 07 /sch ema# ",
"description": "S u mmariz es a b ook th at is stock ed b y th e b ook store...",
"type": "ob ject",
"properties": {
"b ook I d": {
"type": "string "
},
"isb n": {
"type": "string "
},
"title": {
"type": "string "
},
"au th ors": {
"type": "array",
"items": {
"$ ref ": "# /def initions/B ook Au th or"
API Description Formats 245
}
}
},
"def initions": {
"B ook Au th or": {
"type": "ob ject",
"properties": {
"au th orI d": {
"type": "string "
},
"f u llName": {
"type": "string "
}
}
}
}
}
Listing 13.5 An API Profile in XML Using the ALPS Draft 02 Format
< ! - - B ook S u mmary: one or more of th ese may b e retu rned - - >
< descriptor id= "B ook S u mmary" type= "semantic">
< descriptor id= "b ook I d" type= "semantic">
< doc> An internal identif ier, separate f rom th e I S B N, th at identif ies
th e b ook w ith in th e inv entory< /doc>
< /descriptor>
< descriptor id= "isb n" type= "semantic">
< doc> Th e I S B N of th e b ook < /doc>
< /descriptor>
< descriptor id= "title" type= "semantic">
< doc> Th e b ook title, e.g ., A Practical Approach to API D esig n< /doc>
< /descriptor>
< descriptor id= "au th ors" type= "semantic" rel= "collection">
< doc> S u mmariz es a b ook th at is stock ed b y th e b ook store< /doc>
< descriptor id= "au th orI d" type= "semantic">
< doc> An internal identif ier th at ref erences th e au th or< /doc>
< /descriptor>
< descriptor id= "f u llName" type= "semantic">
< doc> Th e f u ll name of th e au th or, e.g ., D . Keith C asey< /doc>
< /descriptor>
< /descriptor>
< /descriptor>
< /alps>
API Description Formats 247
Listing 13.6 APIs.json Example Offering an Indexed View of an API and Its Various
Machine-Readable Description Files
properties:
- type: D ocu mentation
u rl: h ttps://ex ample.com/docu mentation
248 Chapter 13 Document the API Design
- type: OpenAPI
u rl: h ttp://ex ample.com/openapi.json
- type: J S ONS ch ema
u rl: h ttp://ex ample.com/json- sch ema.json
contact:
- F N: API s.json
email: inf o@ apisjson.org
X - tw itter: apisjson
specif ications:
- name: OpenAPI
description: OpenAPI is u sed as th e contract f or all of ou r API s.
u rl: h ttps://openapis.org
- name: J S ON S ch ema
description: J S ON S ch ema is u sed to def ine all of th e u nderlying ob jects
u sed.
u rl: h ttps://json- sch ema.org /
common:
- type: S ig nu p
u rl: h ttps://ex ample.com/sig nu p
- type: Au th entication
u rl: h ttp://ex ample.com/au th entication
- type: L og in
u rl: h ttps://ex ample.com/log in
- type: B log
u rl: h ttp://ex ample.com/b log
- type: Pricing
u rl: h ttp://ex ample.com/pricing
Code examples come in a variety of forms, from just a few lines that demonstrate
how a specific operation works to more complex examples that demonstrate a
complete workflow.
Notice in this example that there is no code to write. The developer only needs to fill
in their API key to obtain a credit card token in their sandbox environment.
Example code that requires developers to write lots of code should be avoided
at this stage to achieve a lower TTFHW. Never require developers to write code
to complete an example when first trying out the API. Instead, make it easy to get
started and see the request work successfully.
values to increase understandability. Choose variable and method names that make
the code easy to read.
Following is an example of charging a credit card using Stripe’s Ruby-based
helper library:
# C reate a C u stomer:
cu stomer = S tripe::C u stomer.create(
:email = > "paying .u ser@ ex ample.com",
:sou rce = > tok en,
)
Note that workflow code examples are more complex than those used to achieve a
quick TTFHW. These examples need to be short enough to explain the concepts but
From Reference Docs to a Developer Portal 251
not too long that they require considerable time to understand. It is often best to
demonstrate scenarios that are easily understood and likely map to customer needs.
the API, and assistance to developers on how to integrate the API. It also provides the
interface on top of the many faceless APIs that exist within an organization’s API
portfolio for evangelism across the organization.
CASE STUDY
Enterprise Developer Portal Success
An API program initiative for a large enterprise IT group started with just
a few key people. After a year of investment, the team had produced several
APIs that offered a number of high-value capabilities to the business. How-
ever, the team produced only reference documentation—no developer por-
tal. As a result, information about how to start using the API wasn’t readily
available. With help, the team expanded the reference documentation into a
complete developer portal.
Their revised developer portal guides developers through an introduction
to the API’s structure and capabilities, onboarding in a sandbox environment
for integration, and production access through a lightweight certification
program.
Influential executives use the developer portal to evangelize the API pro-
gram throughout the organization, resulting in increased demand for adopt-
ing APIs. The developer portal now serves as a central communication tool
and a method of promoting the program to both technical and nontechnical
teams.
Beyond these essential elements, developer portals seek to inform and deliver on
the following experiences:
When discussion with API consumers isn’t possible, try to find other developers
to review the documentation. Conduct a documentation audit by defining a mythical
scenario, then writing some code to call the API to produce a prototype. Along the
way, ask questions to identify areas of improvement for the API documentation
offered.
tools upon them. They should be considered first-class team members rather than
a siloed team that has API implementations thrown at them at the last minute for a
quick-and-dirty documentation effort.
Finally, remember that API documentation is the user interface for developers.
Technical writers can make or break an API’s success. The same can be said for
enterprise API platforms where some APIs are targeting partners, customers, and
third-party service integrators.
Once all items on this list are checked off for all sections, the API developer portal is
in good shape to support the needs of initial consumers involved in the early stages
of API design as well as the needs of future consumers who may discover the API.
Depending on available expertise and the number of API operations, the effort
required to complete this phase may take between one and three weeks. If necessary,
focus on the most common use cases that the API addresses, then incorporate addi-
tional documentation in future phases.
Phase 2: Improvement
The best place to spend time improving the portal will depend on the characteristics
of the API. If the API has changed, has new operations, or works differently than
previously documented, the first priority is updating the docs to incorporate the
changes. But if everything is up to date, consider some of the ideas in Table 13.2 as
time allows.
258 Chapter 13 Document the API Design
• Add case studies: Case studies demonstrate an API’s value by describing how
clients have used it to solve a problem, expand business, or succeed in some
way. They add depth and meaning to API documentation by offering real-
world context, which helps readers understand how the API has already ben-
efited others and could benefit them too. Case studies can even inspire new
ideas for using an API. If “Case Studies” sounds a little dry or academic, try
something like “Success Stories” or “Client Stories” instead.
• Add getting started guides: Readers who understand how APIs work might
be able to start using an API with authentication details alone, but what about
Tools and Frameworks for Developer Portals 259
users who are less comfortable? A getting started guide should build users’
confidence that they can use the API and inspire them to dig deeper into the
rest of the documentation.
• Incorporate analytics: Analytics help portal administrators tailor the portal
to the needs of the audience based on real data about traffic patterns and help
readers move more smoothly through the content.
• Move to single-page format: Consider restructuring some portions of the
portal on a single page. The benefit of this format is that users can navigate the
documentation either with the menu that links to all the section headings or by
using Ctrl/Cmd+F to search for text on the page.
• Translate the documentation: As the API gains traction, consider whether
documentation translation would be helpful. Professional translation is
expensive and takes time, so a clear and persuasive business case is necessary
before starting this journey. It’s rare, but some teams discover that most of
their users are in another country and therefore would benefit from translated
documentation.
Finally, continually check around to see what other companies with successful
APIs are doing with their documentation. Then produce a plan for incorporating
these new ideas into the developer portal to benefit customers, partners, and internal
developers.
• Static site generators: Tools such as Jekyll, used to power GitHub pages, and
Hugo are popular choices for creating and managing developer portals. Pages
are authored using Markdown or similar notation and are stored in a code
repository. Deployment is typically automated to ensure the latest version of
the documentation is published once changes are merged into the main branch.
• SwaggerUI: This is the tool that started it all for the Swagger API description
format, now separated from the tool as the OAS. This open-source codebase
renders any OAS v2 or v3 specification, plus older Swagger specification files,
into API reference documents in HTML format.
260 Chapter 13 Document the API Design
• MVP template: I have collaborated with others to create a GitHub project for
starting an API developer MVP that is based in Jekyll. It helps to combine the
static site generator with some placeholders for content and integrating Swag-
gerUI or similar reference documents into a single location. Fork the reposi-
tory at https://github.com/launchany/mvp-template and customize as needed
to get started quickly.
Summary
Establishing an API documentation strategy is part of delivering a successful API
product, formalized API program, or enterprise API platform. Developer portals
must support a variety of personas. It is critical to ensure that documentation is part
of the overall API design and delivery lifecycle. Otherwise, it becomes a last-minute
task that results in poor documentation that fails to meet the needs of the target
personas.
Seek to include documentation and API portal updates as part of the overall
delivery schedule. An API should be considered done only when the documentation
is updated alongside the release. This approach to documentation will produce a
more complete API that encourages rapid adoption by developers and other decision
makers.
Chapter 14
You have to be really consciously careful about API design. APIs are forever.
Once you put the API out there, maybe you can version it, but you can’t take it
away from your customers once you’ve built it like this. Being conservative and
minimalistic in your API design helps you build fundamental tools on which
you may be able to add more functionality, or which partners can build layers
on top of.
— Werner Vogels
Managing change is not easy, yet it is an inevitable part of maturing an API. For
developers working within a single codebase, change can be difficult but is manage-
able. Refactoring tools and automated test coverage are leveraged to assess the
impact of a change.
When the change involves Web-based APIs, change becomes even more challenging.
Some teams may have a direct relationship with every API consumer, allowing for
changes to be introduced gradually and in coordination with all parties. However,
that is usually not the case. Instead, most consumers of an API have no personal
relationship with the team that owns the API. Extra care is required to manage changes
to an API design to avoid customer churn. This chapter presents some considerations
to determine the impact of change and strategies to introduce change to API designs
that minimize the impact to API consumers.
261
262 Chapter 14 Designing for Change
The fictional online bookstore example used throughout this book assumes
that the APIs identified throughout the process did not already exist, resulting in a
greenfield project. The reality is that organizations already have APIs in production
for a variety of purposes, and any proposed API designs must fit the reality that
brownfield development will be required.
These brownfield initiatives are forced to reconcile the findings from the
ADDR process with any existing API designs to determine the best path forward.
This chapter details some considerations for handling change when APIs already
exist.
Using these factors as a starting point, itemize the differences between existing
APIs and the ideal API design created as a result of the ADDR process. Assign a
sizing for the value provided to API consumers by the new API design and the size
of the impact in API design change. Using t-shirt sizing (e.g., small, medium, large,
extra-large) ensures the measurement is an effective way to size the value and impact.
Then determine what is best for API consumers.
• Who are the API consumers? Internal consumers may offer easier change
coordination, whereas partners may be resistant to making changes to integra-
tions. Customers and third parties acting on behalf of customers may be una-
ble to make changes because of limited or no development resources available.
• What kind of relationship has been established with API consumers? An
internal or external party that the team knows personally can more easily
negotiate for breaking changes. API consumers that have no relationship may
be more challenging. API consumers that are heavy influencers in the market-
place may have a negative impact on current and future customer prospects if
they are cornered into adopting unnecessary API changes.
• What value is being delivered to API consumers as a result of the change?
API changes that improve the use of the API may be well received. Changes
may also unlock new capabilities that consumers have been requesting, even
with the cost of change. For others, it may give them pause to consider moving
to a different vendor, resulting in customer churn.
How an organization manages change with its API consumers tells a lot about
who and what it values. If the API provider prefers to deliver API design elegance at
the cost of constant breaking changes, API consumers may soon start shopping for
alternatives. If, however, the API provider values the API consumer above having the
perfect API design, it may just find itself the leader in the marketplace.
strategy that encourages existing integrations to migrate to the new operations one
at a time.
However, some compromises may be more significant, such as an existing API
design that is too low level. This issue is common for APIs that opt to expose database
tables directly compared to the new proposed API design that would apply a more
coarse-grained design with an outcome-based focus. Mixing low-level and high-level
APIs may create too much cognitive dissonance for developers and therefore are less
than ideal.
Teams must determine if they wish to add the new design to an existing API,
start a new API as if it were a brand-new product offering, or deliver the new design
as a new version of the existing API. Each option will have an impact both on the
organization and on current and future API consumers.
If the existing API design impedes the API consumer’s ability to use the API
effectively, a more greenfield approach may be required. Keep in mind that if the
team chooses to release a new API product or version, additional resources will be
required to maintain both APIs for some amount of time in the future. The next
section discusses API versioning strategies and considerations.
happens, teams should try to ensure that they do not introduce breaking changes that
will force their API consumers to fix code. For some API consumers, updating code to
adapt to an API change may not be an option at all. Therefore, it is important to
understand what may constitute a breaking change, then establish an API versioning
policy that encourages the evolution of an API over time without breaking existing API
consumers.
• Adding a new API operation. Existing client code won’t use the operation, so
no harm is done to existing integration.
• Adding an optional field to a request message. In this case, existing client code
will not be forced to add the new field.
• Adding a required field to a request message with a default value. For cli-
ent code written prior to the addition, the server will apply the default value
because it would be missing from the request.
• Adding a field to a response message. Existing client code should safely ignore
the new field(s) unless they opted to use a mapping library that raises an error
if the newly added field cannot be found in the destination object. This is an
antipattern for API consumption but may be encountered in some circum-
stances, so use caution.
• Adding a value to an enumeration field type. A new enum value that is deserial-
ized on the client may not have a known display string associated to it. To be
a nonbreaking change, older clients must run correctly when receiving a new
enum value. Not all clients may be designed in this way, so caution is advised.
Incompatible Changes
Changes that are incompatible with existing integration code include, but are not
limited to, the following:
• Renaming fields and/or resource paths, as existing client code will require a
code change to adapt to the renamed value
• Renaming or removing fields in a request or response
• Removing API operations used by existing API client code
266 Chapter 14 Designing for Change
Remember that once an API is released into production and has at least one
integration, the design decisions are permanently a part of the API. This is why
the ADDR process is so important—it helps teams to validate design decisions
before an API goes into production. However, a proper API versioning strategy
can assist in mitigating some of these issues while allowing an API design to
evolve over time.
POST /v1/calculate-sales-tax
POST /v2/calculate-sales-tax
Deprecating APIs
Nothing will ruin a team’s week more than scrambling to find a replacement to an
API that has been shut down overnight. To avoid this kind of impact on existing API
consumers, teams must define their deprecation policy and communicate it to their
API consumers.
Deprecating APIs 269
Announcing a Deprecation
Communicating a deprecation is a significant factor in maintaining API consumer
trust. The methods of communication vary, but should include
Applying an API stability contract gives API providers the freedom to introduce
new API operations or experimental APIs early for design feedback prior to support-
ing it on a long-term basis.
Summary
Changes to API design cannot be avoided. Whether internal or external, consumers
depend on the API to remain stable in the face of improvements. Introducing changes
to an API’s design provides an opportunity for the owning team and the organization
to maintain the trust of their API consumers. By applying an appropriate API
versioning strategy, taking appropriate steps to deprecate APIs when they are no
longer needed, and establishing an API stability contract, teams are able to manage
change while avoiding negative impacts to their API consumers.
This page intentionally left blank
Chapter 15
Protecting APIs
API design doesn’t stop at HTTP methods, paths, resources, and media types. Pro-
tecting APIs from malicious attackers is an essential part of API design. If left unpro-
tected, an API will become an open door that can do irreparable damage to an
organization and its customers. An API protection strategy involves the implementa-
tion of the right components, selection of an API gateway solution, and integrating
an identity and access management to tie it all together.
This chapter outlines some foundational principles and provides guidance
on common practices along with antipatterns to avoid when approaching an API
protection strategy. Resources are provided for further reading and research on the
journey.
• Gaining access to a user database via an unsecured API, allowing the bad guy
to confirm the identities of 15 million accounts on Telegram while remaining
undetected.
273
274 Chapter 15 Protecting APIs
• Exploiting a password reset API that returns the reset token, allowing the con-
firmation email to be bypassed and accounts to be taken over, exposing sensi-
tive health and personal details.
• Combining large data sets from previous hacks to confirm authorization of
users, resulting in the ability to pass security screening and download tax
returns from the US Internal Revenue Service.
• Reverse-engineering undocumented APIs intended for internal, private use by
a company for its mobile apps, allowing the bad guy to access data easily with
minimal or no implemented protective measures. This security risk is common
for many API vendors that consider an undocumented API as secure, such as
Snapchat.
• Exposing the exact location, by latitude and longitude, of users because a
previously private Tinder API was opened for end users. A thorough security
review prior to opening the API to developers would have identified that the
mobile app, not the API, was responsible for hiding the actual physical loca-
tion of their users.
These recent breaches span from low-reward results, such as disclosing business
intelligence as a competitive advantage, to high-reward results that can disclose
extremely sensitive data. One even jeopardized the safety of individuals by disclosing
their exact location!
Unfortunately, some API providers may take shortcuts in securing their internal
APIs. Perhaps they mistakenly think that if they do not document the potential access
to the API, no one will go looking for it. This misguided belief is naïve at best and risks
exposing the organization to various attack vectors that it could otherwise avoid.
• Data scraping and botnet protection: Detects intentional data scraping via
APIs, online fraud, spam, and distributed denial-of-service (DDoS) attacks
from malicious botnets. These attacks tend to be sophisticated and require
specialized detection and remediation.
• Review and scanning: Manual and/or automated review and testing of API
security vulnerabilities within source code (static reviews) and network traffic
patterns (real-time reviews).
Not all of these practices are included in a single solution. Instead, several
components must be considered as a necessary part of an API protection strategy.
API Gateways
API gateway is both a pattern and a classification of middleware. The API gateway
pattern involves the addition of an extra network hop that the client must traverse to
access the API server.
API gateway middleware is responsible for externalizing APIs across network
boundaries. They may act as a pass-through or perform protocol transformation as
part of the process. The API gateway becomes a central gatekeeper for all traffic in
and out of the API.
API gateway middleware may be standalone products or a component within a
larger product offering, such as an API management layer. While API gateways may
be built from the ground up, some gateways are composed from building blocks such
as a reverse proxy and plug-ins to realize the features needed. API gateways rarely
address more advanced features needed to manage APIs as products. These concerns
are offered by API management layers.
API Management
API management layers, or APIMs, include an API gateway but also extend their
capabilities to include a complete API lifecycle management solution. The solution
includes publishing, monitoring, protecting, analyzing, and monetizing APIs. It may
also include community engagement features.
Components of API Protection 277
Service Meshes
Service meshes shift the needs of network reliability, observability, security,
routing, and error handling away from each process to separate out-of-process
infrastructure. This new infrastructure is portable and independent of any specific
programming languages and frameworks selected by each process, making it
portable. Service meshes have grown in popularity due to the introduction of
microservices but may be used for any architecture or combination of architectural
styles.
Service meshes replace the direct communication of processes with a series
of proxies that direct the communication and error handling on behalf of the
process. A proxy is deployed alongside each running process to eliminate any
central point of failure. Deployment is often to a single virtual machine (VM)
or alongside each container as a sidecar. A centralized management control
plane is used to configure the proxies, communicate outages, and oversee the
network health. The controller, however, does not involve itself with network
data communications.
The components of a service mesh are shown in Figure 15.1.
Service meshes may be seen as a competitor to API gateways and APIMs. How-
ever, this is not the case. While service meshes manage on OSI layer 4 (TCP/IP) and
OSI layer 7 (HTTP), they are often paired with an API gateway or APIM. This offers
the best of both worlds by providing resilient, observable network communications
using a service mesh, with API product and lifecycle management offered by an
APIM or API gateway.
Service meshes introduce additional network hops and therefore may have a nega-
tive impact on network performance. However, the capabilities offered by a service
mesh may offset the negative impact and may produce a net gain when factoring in
the many separate network management elements that have to be coordinated when
a service mesh is not present.
Finally, bear in mind that smaller organizations may not see the need for the
added complexity of a service mesh. However, larger organizations managing many
developer teams producing a multitude of services across one or more cloud environ-
ments may benefit from the use of a service mesh.
278 Chapter 15 Protecting APIs
Figure 15.1 The components of a service mesh, including the proxy instances, each
connected to a central control plane for oversight and configuration.
API Proxy
https://api.example.com
Your API
Reporting
API Management
Your
Platform
Infrastructure
Infrastructure
a large number of API requests, with a variety of subscription levels and service-level
agreements (SLAs) offered to customize the solution. Hosting an APIM is a great
option for smaller organizations or for organizations beginning to embark on the
API journey. However, they may become costly and are often moved on-premises as
the API program matures. Figure 15.2 illustrates the hosted APIM option.
On-premises APIMs are installed in a data center or cloud infrastructure. They
place more burden on the operations teams to ensure proper reliability and availabil-
ity than hosted APIMs but also offer more customization options. In addition, on-
premises installations allow organizations to install multiple instances of the gateway
to isolate APIs involved in regulatory audits or to isolate the impact of API usage
across partners and customers. They are also useful when API gateways are desired
to manage internal-facing APIs that are not externalized to the public Internet.
Figure 15.3 illustrates the on-premises APIM option.
The third type of APIM management option is hybrid. Hybrid installations use a
hosted dashboard and reporting infrastructure offered by the vendor while support-
ing API gateway instances to be deployed using an on-premises model. This is the
option that is seen least in the field. The primary advantage is to reduce the burden
of supporting the various processes involved in analysis and reporting systems, par-
ticularly if the organization lacks in-house expertise for some of the related compo-
nents or database vendors. Figure 15.4 illustrates the hybrid APIM model.
Keep in mind that some cloud infrastructure providers offer their own API gate-
way or APIMs. While this may be useful in the short term, some organizations
may find the customization effort required to be too great. Organizations that are
required to take a multicloud approach may opt to select a third-party APIM vendor
rather than try to support multiple cloud-provided API gateways. Whatever the case,
select the best fit for the current stage of the API program, reevaluating to ensure the
best option continues to be in use.
API Gateway Topologies 281
API Proxy
https://api.example.com
Your API
Reporting
Your
Infrastructure
Figure 15.3 The on-premises API management option.
API Proxy
https://api.example.com
Your API
Reporting
API Management
Your
Platform
Infrastructure
Infrastructure
Figure 15.4 The hybrid API management option.
1. Scott Rose, Oliver Borchert, Stu Mitchell, and Sean Connelly, Zero Trust Architecture (National Insti-
tute of Standards and Technology (NIST) Special Publication 800-207, August 2020), https://nvlpubs.
nist.gov/nistpubs/SpecialPublications/NIST.SP.800-207.pdf.
API Gateway Topologies 283
Web App
API Server
Cluster
Partner
Integration
Partner
Integration
/orders/{orderId}/payments
Payment Service
Figure 15.6 API topology 2 showing an API gateway routing to multiple backend services
based on the base path of the incoming API request.
may be dedicated to one or several tenants of a multitenant SaaS. Issues with avail-
ability of one gateway instance should not negatively impact the other gateway
instances, limiting the impact during peak usage scenarios. This topology is shown
in Figure 15.7.
/carts
Shopping Cart
Web App Service
Public
API Gateway
Mobile App
/orders
Create Order
Service
Web App
PCI-Compliant
API Gateway
Mobile App
Payment Service
/orders/{orderId}/payments
Figure 15.7 API topology 3 showing multiple API gateway instances that support various
internal and external API clients, including the isolation of payment processing for PCI
compliance and auditing.
• Passwords are fragile because they change often, which would render any code
unable to use the API until it is updated with the new password.
• Delegating access to some or all data to third parties requires sharing the
password with them.
• The use of username and password does not support multifactor authentication.
286 Chapter 15 Protecting APIs
To avoid these challenges, the use of API keys or API tokens is preferred for
most situations. These two concepts are often used interchangeably but are quite
different.
API keys are simple replacements for a password and have no expiration date.
They are often found in a user profile page or in the settings page for a Web applica-
tion. An API key may be a long alphanumeric value (e.g., l5v z a8 u a8 9 6 max h m). Because
API keys have no expiration date assigned, anyone who obtains the key maliciously
may be able to use the API to access data and backend systems for an indefinite
period of time. Resetting an API key usually requires a manual step within the same
user profile or settings page, assuming that the API provider offers API key reset
capabilities at all.
API Tokens
API tokens are a robust alternative to API keys. They represent a session where a user
is authorized to interact with an API. While they may be alphanumeric and look
similar to an API key, they are not the same. An API token may represent a user or a
third party who has been given limited or full access to the API on the user’s behalf.
API tokens also have an associated expiration time.
An API token’s expiration time may vary from a few seconds to a few days
depending on various configuration elements. With an API token also comes a refresh
token, which allows the API client to request a new API token when the previous one
has expired or is no longer valid.
An API token may have one or more access controls associated with it. These
controls are often referred to as scopes. Multiple API tokens may be generated
on behalf of a user, including one with an assigned scope for read-only access of
a specific API resource, another with assigned scopes for read/write access to all
resources, and yet another that offers a single scope assignment for limited API
resource access by a delegated third-party application. API tokens are illustrated
in Figure 15.8.
APIs often use a variety of methods for passing an API token to the server, includ-
ing as a query argument on the URL, as a POS T parameter, and through an HTTP
header. Avoid using query arguments in the URL, as the API token will be logged by
Web servers and reverse proxy servers, and JavaScript code may also be allowed to
access the API token easily. POS T parameters tend to be more secure, but the location
of the token will vary across APIs.
Therefore, it is recommended to use the standardized HTTP Au th oriz ation
header. Access to HTTP headers can be limited through the use of CORS response
headers, and headers are less likely to be logged by intermediary servers.
Identity and Access Management 287
Web App
Token Expired? No
Required scopes supplied? Yes
Mobile App
API Gateway
Token Expired? No
X API
Server
Required scopes supplied? No
Mobile App
Token Expired? Yes X
Required scopes supplied? Yes
Figure 15.8 Three separate API tokens, only one of which is valid and allowed to pass to
the API server by the API gateway.
It is the responsibility of the API server to dereference the API token to determine
the specific user making the API call, along with any other details.
Pass-by-value API tokens contain name/value pairs included within the token.
This reduces the number of lookups required to dereference a token to its associated
values by the API server.
API tokens that use pass by value typically allow the API client to access the same
name/value pairs that are available to the API server. Therefore, pass-by-value API tokens
should embed feature flags or other sensitive data that could be used to compromise a
system. Instead, use them to convey minimal details, such as opaque identifiers.
288 Chapter 15 Protecting APIs
A popular pass-by-value API token standard is the JSON Web Tokens (JWTs),
typically pronounced “jot.” JWTs are composed of three elements: a header, pay-
load, and signature. Each element is Base64 encoded and dot-separated to compose
an opaque token that may be used as an Authorization bearer token between client
and API. JWTs are signed to ensure they haven’t been tampered with by the client
before being sent to the server. Using a private key signature provides further protec-
tion against tampering and verifies the identity of the client. The JWT.io2 Web site is
an excellent resource for learning more about JWTs.
JWTs tend to be more popular for communicating authorization details for east–
west traffic, while pass-by-reference API tokens are used for north–south traffic.
2. https://jwt.io
3. https://oauth.net/2
Considerations before Building an In-House API Gateway 289
API API
User Web App
Gateway Server
Third-Party
IAM Server
Authorization Server
Figure 15.9 The core components and basic interaction of OAuth 2.0.
clients to verify the identity of the end user as well as to obtain basic profile informa-
tion using a REST-like API. Without this protocol, custom integration is required to
bridge identity and profile details between the authorization server and the API. The
specification details, along with an updated list of OpenID Connect–compliant serv-
ers, is available on the OpenID Connect4 Web site.
Enterprises that require federated identity management across multiple internal
and third-party vendors lean heavily on single sign-on (SSO) for their Web applica-
tions. Security Assertion Markup Language (SAML) is a standard used to bridge
APIs into an existing SSO solution within the enterprise, making the transition bet-
ter for enterprise users accessing an API through an application. More details are
available on the OASIS SAML5 Web site.
4. https://openid.net/connect
5. https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=security
290 Chapter 15 Protecting APIs
Summary
API design requires considering how an API will be protected from malicious
attackers. Unprotected APIs are an open door that welcome attackers to damage an
organization and its customers. An API protection strategy involves the
implementation of the right components, selection of an API gateway solution, and
integrating an identity and access management to tie it all together.
Don’t leave API protection to someone’s side project or to a well-intentioned
team within the organization. Select the right approach with vendor-supported
components that ensure that the front door of the organization’s APIs is barred shut
rather than left unlocked.
This page intentionally left blank
Chapter 16
When done effectively, governance can provide clear direction, remove obstacles,
and allow different parts of the organization to function independently.
— Matt McLarty
An organization that produces more than one API product must learn to scale its API
design process. Otherwise, designs will lack consistency across the portfolio of APIs
produced by the organization. Authentication and authorization will vary between
APIs. Naming conventions and error responses will deviate. In short, the API pro-
gram will become a mess.
This chapter explores the factors required to scale API design efforts within
an organization. These factors include establishing a style guide for consistency,
incorporating design reviews, and encouraging a culture of reuse. Once these measures
are applied, teams will be able to function independently while maintaining consistency
across the API portfolio. Finally, the chapter takes a look back at the topics covered in
this book and offers some guidance on how to continue the API design journey.
293
294 Chapter 16 Continuing the API Design Journey
• Introduction: The scope of the style guide, who to contact for questions,
clarifications, or enhancements
• API fundamentals: Used to educate and coach those less familiar with the
basics; may consist of links to internal or external training materials
• Standards: Naming conventions, guidance for selecting HTTP methods and
response codes, organizing resource paths, resource lifecycle design, payload
and content formats, when and how to use hypermedia
• Design patterns: Common patterns encountered, including pagination, error
responses, bulk processing, singleton resources
• Lifecycle management: Recommendations for moving an API into production,
along with deprecation and sunset procedures
• Tools and technologies: List of tools that are recommended, including those
with site licenses already available
• Operational recommendations: Recommended API management tools, con-
figurations, processes, marketing recommendations, and common practices
for highly available, robust, and resilient APIs
• Further reading: Additional resources that may be interesting to the reader,
including both internal and publicly available papers, articles, and videos
Too often, style guides are used to push an agenda. Full compliance, or else. That
isn’t what style guides should be about. Their goal should be to advise teams designing
APIs toward a more consistent API with other APIs across the organization. A newly
hired developer should be able to work with a variety of APIs across the organization
without realizing that different teams designed them.
access to shared services and support (e.g., API management layer, opera-
tional and infrastructure support) rather than being forced to implement it
themselves.
2. Federated: A centralized team oversees and maintains the style guide, but
coaches, embedded locally within the business unit and/or region, are available
to address their specific needs. This method avoids the ivory tower problem of
a committee designing standards without understanding the needs of specific
business units.
3. Clone and customize: A single group manages the style guide. Teams clone
the standards as a starting point, making minor enhancements locally for
business unit consistency. For organizations that have many independent teams
within and/or across business units, this is the most effective method.
• Will the organization be enforcing the standard? If so, then use RFC 2119 rec-
ommendations to enforce what MUST, SHOULD, and MAY be implemented.
• Will enforcement be deferred to a future date? Then go ahead and start using
RFC 2119, but keep the wording to lowercase (e.g., must, should, may) until it
is enforced. This demonstrates expectations and likely future enforcement but
with less of a formality during the initial introductory period.
• Is the guide shared across business units, limiting the organization’s ability to
control or strictly enforce the guidelines? Then soften the tone and focus on
design consistency by encouraging teams to adopt as many of the guidelines as
possible rather than using a more formal tone.
1. S. Bradner, “Key Words for Use in RFCs to Indicate Requirement Levels,” March 1997, https://
datatracker.ietf.org/doc/html/rfc2119.
296 Chapter 16 Continuing the API Design Journey
Remember
The goal of an API style guide is to advise teams designing APIs toward
consistency with other APIs across the organization.
2. API Stylebook: Collections of Resources for API Designers, maintained by Arnaud Lauret, accessed
August 24, 2021, http://apistylebook.com.
Conducting API Design Reviews 297
that there is significant deviation in standards and common practices across API
styles. Remember that it is better to follow common practices for each API style than
to try to unify all API styles into a single set of recommendations.
Finally, keep in mind that there is a cost to supporting each API style. Take time
to understand the needs for the new API style. Then determine if the needs outweigh
the cost required to build and support yet another API style guide.
Following are some tips and insights on conducting healthy API design reviews.
• API name: The name should be descriptive and make it easy to determine the
scope of the API when first discovered.
• API description: The description should be comprehensive, starting with an
overview of the API and including a list of use cases it solves.
• API operations: Each operation should offer a summary of what task, activity,
or outcome it produces along with a description that includes detailed usage
instructions. Ensure all input and output values are captured and properly
described, including expected formats that could cause errors if violated.
• Example usage: Examples of API usage are often the most important, yet
missing, element of an API’s documentation. These examples do not need to
be in a specific programming language (although that helps when trying to
offer an API to a broad audience). Simple HTTP request/response examples,
perhaps complemented with Postman collections, will go a long way toward
accelerating developer understanding and completing the integration effort.
• Avoid internal references: Great documentation assumes that readers have no
idea about any of the internal systems or implementation choices behind the
scenes. They just want to get something done and they want to find out if the
API will help them achieve their goals.
Conducting API Design Reviews 299
other description formats to produce a mock version of the API design. While the
mock API won’t be fully formed, it will provide a basic understanding of how the
API will be used once completed and catch suboptimal design decisions early.
Consumption Goal
Onboarding Register for portal and API access
Discovery Identify API capabilities
Mapping Map solution to platform API capabilitics using reference documentation
Exploration Prototype consumption (“try-it-out”)
Integration Consume via code
Certification Obtain approval for production API access
Usage monitoring Production access monitoring and throttling for compliance
Platform improvement Request platform API enhancements to meet the needs of the solution
Platform updates Update notifications for new API endpoints, enhancements, case studies
Figure 16.1 The API consumption lifecycle, showing the phases that a development team
experiences when they find a new API.
The Journey Has Only Begun 301
modeling prior to designing the API. The API is then designed and refined through
feedback with those who will integrate the API into their solution.
While some may think that the journey has been completed, this is only the
beginning. The API design will now be delivered and managed. It will meet real-
world usage, perhaps even encountering new use cases never considered. The
ADDR process will be used once again as the API grows and matures. For larger
organizations, this API design lifecycle will be repeated for many new APIs, requiring
the ADDR process to be scaled for use by multiple teams. The journey has only
begun.
Appendix
HTTP Primer
To better understand how Web APIs work, it is important to start with an under-
standing of HTTP, the language of the Web. While the HTTP protocol can be hid-
den behind various libraries and frameworks, understanding the protocol provides a
foundation for troubleshooting API integrations and improved API design.
This primer offers an introduction to the HTTP protocol, the elements that are
involved in using HTTP for interacting with Web APIs, and some advanced features
that help to shape more powerful API interactions.
Overview of HTTP
The HTTP protocol is a client/server protocol. An HTTP client sends a request to a
server. The HTTP server then determines if it can service the request with the
information given. The server then returns a response that includes a code indicating
success or failure, along with a response payload containing the information
requested or details about the error. This request/response flow is illustrated in
Figure A.1.
303
304 Appendix
HTTP/1.1 200 OK
{
"id": "12345",
"projectName": "My Project"
…
}
• https://www.google.com
• https://launchany.com/effective-api-programs/
• https://deckofcardsapi.com/api/deck/new/shuffle
• Protocol: The protocol used to connect (e.g., http [unsecure] or https [secure]).
• Hostname: The server to contact (e.g., api.example.com).
• Port number: A number ranging from 0 to 65535 that identifies the process on
the server where the request is to go (e.g., 443 for https or 80 for http).
• Path: The path to the resource being requested (e.g., /projects). The default
path is /, which indicates the homepage.
• Query string: Contains data to be passed to the server. Starts with a ques-
tion mark and contains name= v alu e pairs, using an ampersand as a separator
between them (e.g., ? pag e= 1& per_ pag e= 10).
HTTP Request 305
Port
HTTP Request
An HTTP request is composed of several parts: the HTTP method, the path, the
header, and the message body.
The HTTP method informs the server what kind of interaction the client would
like to request. Common HTTP methods are GET, to request data, and POS T to submit
data. The methods commonly used for Web-based APIs are detailed later in this
appendix.
The path is the portion of the URL that references a resource on the server. The
resource may be a static file, such as an image, or a piece of code that performs
dynamic request processing.
The header tells the server about the client and specifics about the request. The
header is comprised of header fields in name:v alu e format. Common HTTP request
headers used with Web-based APIs include the following:
• Accept: Informs the server what content types the client is able to support.
Examples may include imag e/g if and imag e/jpeg . If the client is willing to
accept any kind of response, * /* is used. This header is often used with content
negotiation, detailed later.
• Content-Type: Informs the server the content type of the request message
body. Used when submitting data using a HTTP method that requires a mes-
sage body (e.g., POS T).
• User-Agent: Provides a free-form string indicating the kind of HTTP client
that is making the request. This may indicate a specific browser type and version
or may be customized to indicate a specific helper library or command-line tool.
306 Appendix
The message body provides details to the server when data is being submitted and
may be human-readable or binary, as required by the server. For a retrieval request
using GET, the message body may be empty.
Figure A.3 shows an example of an HTTP request sent to Google to request the
homepage that contains the search form, documented line by line.
HTTP Response
Once the request is received by the server, the server processes the request and sends a
response. The response is composed of three parts: the response code, the response
header, and the response body.
The response code is a number that corresponds to a success or error code
indicating whether the request could be fulfilled. The response code sent must be
one of the those outlined in the HTTP specification and are detailed later. Only one
response code is allowed per response.
The response header tells the client specifics about the result of the request. The
header is comprised of header fields in name:v alu e format. Common HTTP response
headers used with Web-based APIs include the following:
• Server: A string that provides details about the vendor and version of the
server that processed the request (e.g., ng inx /1.2.3). The server may choose
to provide little or no detail to avoid exposing details that might indicate a
possible vulnerability exists.
The response message body provides the content back to the client. It may be an
HTML page, an image, or data in XML, JSON, or another format, as indicated by
the C ontent- Type response header.
Figure A.4 shows an HTTP response sent back from Google based on our earlier
request for the homepage.
It is important to note that the response in Figure A.4 includes only the HTML
in the response and not additional images, stylesheets, JavaScript, and so on. The
HTTP client is responsible for parsing the HTML, identifying the tags that reference
these additional assets, and sending subsequent HTTP requests for each one. For a
Web page with 20 images, 21 separate HTTP requests are required to gather all of
the files necessary to render the Web page—one request for the HTML page, along
with the 20 requests necessary to retrieve each image.
• POST: Submits data to the server, often for storage or for calculations—
response not cacheable
• PUT: Submits data to the server, often as a replacement of existing data—
response not cacheable
• PATCH: Submits data to the server, often as a partial update of existing data—
response not cacheable
• DELETE: Deletes an existing resource on the server—response not cacheable
HTTP methods have additional semantics that are important for clients to take
into consideration: safety and idempotence.
A safe method indicates that the HTTP method used will not generate side
effects, such as altering data. This is common for GET and HEAD methods, as they
are intended for resource retrieval and do not alter data. APIs that implement data-
altering operations using safe HTTP methods risk generating unpredictable results,
especially when middleware servers, such as caching servers, are involved.
Idempotent methods ensure that the same side effects are produced when identical
requests are submitted. This is true for GET and HEAD retrieval methods because no
data is altered. PU T and D EL ETE are guaranteed by the HTTP specification to be
idempotent, as PU T replaces the resource with a completely new representation and
D EL ETE removes the resource from the server.
POS T is not guaranteed to be idempotent, as it may create new resources on each
subsequent request or alter data in some way that is not guaranteed to produce the
same results (e.g., incrementing a value). Likewise, PATC H is not idempotent, as only a
subset of fields, rather than the entire representation, is altered.
Figure A.5 summarizes the semantics of the common HTTP methods used for
Web-based APIs.
Figure A.5 Common HTTP methods used with APIs, including safety and idempotency
traits that help to guide the client on how to recover from errors.
HTTP Response Codes 309
Table A.1 offers a list of the common response codes from the HTTP specification
that are used by REST-based APIs.
Content Negotiation
Content negotiation allows clients to request one or more preferred media type(s) for
the server response. With content negotiation, a single operation may support
different resource representations, including CSV, PDF, PNG, JPG, SVG, and others.
The client requests the preferred media type using the Accept header. This example
demonstrates an API client requesting a JSON-based response:
More than one supported media type may be included in the header, as shown in
this example:
The asterisk may be used as a wildcard when selecting media types. A tex t/*
indicates that any subtype of the text media type is acceptable. Specifying a value
of * /* indicates that the client will accept any media type in the response. This is a
common scenario for browsers, which will prompt the user whether to save the file or
launch a chosen application when encountering an unknown media type. However,
for clients working with an API, it is important to be explicit to avoid runtime errors
that could occur when encountering an unknown or unsupported content type.
Requests may specify preference for specific media types supported within the Accept
header through the use of quality factors. Quality factors are expressed as a qvalue
between 0 and 1 that helps to assign a preferred order of media types. The API server
reviews the header values and return the response using the content type that matches
both what the server supports and what the client requested. If the server cannot respond
with an accepted content type, it returns a 415 Unsupported Media Type response code.
Cache Control 311
Here is an example of using qvalues to specify a preference for XML, with JSON
also supported if XML is unavailable:
The use of qvalues allows API client code to support a specific type, perhaps XML
for improved transformation capabilities and JSON as a fallback.
Because API clients may specify more than one media type, they must pay
special attention to the C ontent- Type response header to determine which parser is
appropriate. The following is a response that provides XML based on the previous
example request:
HTTP/1.1 200 OK
D ate: Tu e, 16 J u ne 2015 06 :57 :43 GMT
C ontent- Type: application/x ml
Content negotiation extends the media type support of an API beyond a single
type, such as JSON or XML. It allows some or all operations of an API to respond
with the content type that best meets the needs of the API client.
Likewise, language negotiation allows APIs to support multiple languages in a
response. The approach is similar to content negotiation using the Accept- L ang u ag e
request header and C ontent- L ang u ag e response header.
Cache Control
The fastest network request is the one that doesn’t need to be made. A cache is a
local store of data to prevent re-retrieval of the data in the future, thereby optimizing
network communications. Developers familiar with the term have likely used server-
side caching tools such as Memcached to keep data in memory and reduce the need
to fetch unchanged data from a database to improve application performance.
HTTP cache control allows for cacheable responses to be stored locally by
API clients or intermediary cache servers. This moves the cache closer to the API
client and reduces or removes the need to traverse the network all to the way to
the backend API server. Users experience better performance and reduced network
dependence.
312 Appendix
HTTP makes available several caching options through the C ach e- C ontrol
response header. This header declares whether the response is cacheable and, if so,
for how long it should be cached.
Here is an example response from an API operation that returns a list of projects:
HTTP/1.1 200 OK
D ate: Tu e, 22 D ecemb er 2020 06 :57 :43 GMT
C ontent- Type: application/x ml
C ach e- C ontrol: max - ag e= 240
In this example, the max age indicates that the data may be cached for up to 240
seconds (4 minutes) before the client should consider the data stale.
APIs may also explicitly mark a response as not cacheable, requiring a new request
each time the response is required:
HTTP/1.1 200 OK
D ate: Tu e, 22 D ecemb er 2020 06 :57 :43 GMT
C ontent- Type: application/x ml
C ach e- C ontrol: no- cach e
Applying thoughtful use of the cache control header to APIs reduces network
traffic and speeds up Web and mobile applications. It also is the building block for
conditional requests.
Conditional Requests
Conditional requests are a lesser known but powerful capability offered by HTTP.
Conditional requests allow clients to request an updated resource representation
only if something has changed. Clients that send a conditional request will either
receive a 304 Not Modif ied response if the content has not changed or a 200 OK
response along with the changed content.
There are two precondition types for informing the server about the client’s local
cached copy for comparison: time-based and entity tag–based preconditions.
Conditional Requests 313
Time-based preconditions require that the client store the L ast- Modif ied response
header for later requests. The I f - Modif ied- S ince request header is then be used to
specify the last modified timestamp that the server will use to compare against the
last known modified timestamp to determine if the resource has changed.
Following is an example of a client/server interaction that uses the last modified
date in a subsequent request to determine if the resource has changed on the server:
HTTP/1.1 200 OK
D ate: Tu e, 22 D ecemb er 2020 06 :57 :43 GMT
C ontent- Type: application/x ml
C ach e- C ontrol: max - ag e= 240
L ocation: /projects/12345
L ast- Modif ied: Tu e, 22 D ecemb er 2020 05:29 :03 GMT
The entity tag, or ETag, is an opaque value that represents the current resource
state. The client may store the ETag after a GET, POS T, or PU T request, using the value
to check for changes via a HEAD or GET request.
An ETag is a hashed value of the entire response. Alternatively, servers may
provide a weak ETag, which is semantically equivalent but perhaps not an exact byte-
for-byte equivalency.
Here is a client/server interaction but using ETags rather than the last modified
date:
HTTP/1.1 200 OK
D ate: Tu e, 22 D ecemb er 2020 06 :57 :43 GMT
C ontent- Type: application/x ml
C ach e- C ontrol: max - ag e= 240
L ocation: /projects/12345
ETag : “ 17 f 0f f f 9 9 ed5aae4edf f dd6 49 6 d7 131f ”
HTTP/1.1 200 OK
D ate: Tu e, 22 D ecemb er 2020 07 :33:04 GMT
C ontent- Type: application/x ml
C ach e- C ontrol: max - ag e= 240
L ocation: /projects/12345
ETag : “ b 252d6 6 ab 3ec050b 5f d2c3a6 26 3f f af 51db 10f cb ”
Conditional requests reduce the effort required to validate and refetch cached
resources. ETags are opaque values that represent the current internal state, whereas
last modified timestamps rather than ETags may be used for time-based comparison.
They may also be used for concurrency control when making modifications to
resources.
HTTP/1.1 200 OK
D ate: Tu e, 22 D ecemb er 2020 07 :33:04 GMT
C ontent- Type: application/x ml
C ach e- C ontrol: max - ag e= 240
L ocation: /projects/12345
ETag : “ b 252d6 6 ab 3ec050b 5f d2c3a6 26 3f f af 51db 10f cb ”
PU T /projects/1234
I f - Match : “ b 252d6 6 ab 3ec050b 5f d2c3a6 26 3f f af 51db 10f cb ”
HTTP/1.1 200 OK
D ate: Tu e, 22 D ecemb er 2020 08 :21:20 GMT
C ontent- Type: application/x ml
C ach e- C ontrol: max - ag e= 240
L ocation: /projects/12345
ETag : “ 1d7 209 c9 d54e1a9 c4cf 7 30b e411ef f 1424f f 2f b 6 ”
PU T /projects/1234
I f - Match : “ b 252d6 6 ab 3ec050b 5f d2c3a6 26 3f f af 51db 10f cb ”
The second API client that received the failed precondition response must now
refetch the current representation of the resource instance, inform the user of the
changes, and request whether the user wishes to resubmit the changes made or leave
it as-is.
Concurrency control may be added to an API through HTTP preconditions in
the request header. If the ETag/last modified date hasn’t changed, then the request is
processed normally. If it has changed, a 412 response code is returned, preventing the
client from overwriting data as a result of two separate clients modifying the same
resource concurrently. This is a powerful capability built in to HTTP, preventing the
need for teams to invent their own concurrency control support.
Summary 317
Summary
HTTP is a powerful protocol with a robust set of capabilities, including some that
are less known. Using content negotiation allows API clients and servers to agree on
a supported media type. Cache control directives provide client-side and intermediary
caching support. HTTP preconditions can be used to determine if expired caches are
still valid while protecting resources from overwriting changes. By applying these
techniques, teams are able to build robust APIs that drive complex applications in a
resilient and evolvable way.
This page intentionally left blank
Index
319
320 Index
API stability contract, establishing, software design and, reviewing principles of,
270–71 7–10
API versioning strategies, 264–68 API designers and architects, 27
deprecating APIs, 268–70 API design-first approach, 20–23
determining what is best for API agility of, 22–23
consumers, questions for, 262–63 phases of, 20–21
impact of, on existing APIs, 261–64 principles relevant to concerns of, 22
management built on trust, 264 API design gap analysis, 262
perform API design gap analysis, 262 API documentation, 233–60
strategies for, 263–64 in ADDR process, 24, 25
collaborative, 13 API description formats, 234–48
as communication, 6–7 areas of improvement, questions
data model exposed as, 11 to identify, 253–56
developer experience in, 5 async APIs, 184–85
documentation, 5, 7, 14 developer portals, 251–53
elements of, 4–6, 14, 102 extending docs with code examples,
248–51
outcome-based focus of, 14
helper libraries, 221
principles of, 13–14
importance of, 234
process, 15–28
as most important user interface for
ADDR in, 23–26
developers, 14, 234, 301
API design antipatterns in, 16–20
MVP, 256–59
API design-first approach in, 20–23
in query-based design process, 154–57
applying effectively, 28
REST API design, 118, 120–23
applying product thinking to, 5
review, 298
communication in, 6–7, 15
role of technical writer in API docs, 255–56
DDD in, 26
in RPC API design process, 145–46
reasons for, 15–16
API fundamentals, in API style guide, 294
roles involved in API design sessions, 27
API gateways, 276
product thinking and, 4–5, 6 direct to API server, 283
refined in ADDR process, 24, 25 in-house, 289–91
resiliency to change, evolutionary approach management hosting options, 279–81
for, 14, 264, 301 middleware, 276
resource-based, 10–11 multicloud API management retail (case
reviews, conducting, 297–300 study), 281–82
automated test coverage, 299 multiple instances of, 283–84, 285
benefits of, 297 network traffic considerations, 282–83
caution about, 297–98 routing to services, 283
documentation review, starting topologies, 279–84
with, 298 API keys, 285–86
standards and design consistency, API management layers (APIMs), 260,
checking for, 299 276–77
try it out support, adding, 299–300 APIMatic, 223
scaling, within organization, 293–302 API modeling, 79–98
API consumption lifecycle, 300 API priority and reuse, 95–96, 97
API style guide, establishing, 293–97 API profile structure, 81, 82
culture of reuse, developing, 300–301 defined, 80–81
Index 321
U boundaries, 12, 71
customer- and partner-facing, 112
Uber Engineering, 203 evolvable, 103
Ubiquitous language, 49 high cohesion and loose coupling in, 10
UI (user interface) tests, 228–29 information hiding, 9
Uniform interface, 103 message-based, 12
Unique name, 10 REST-based, 107, 112, 161–62
UNIX, 118, 164 Web application firewalls (WAFs), 277, 278
Unsafe HTTP operation, 93, 115 Webhooks
Unused API antipattern, 20 dispatcher, 171, 172
URI-based versioning, 267–68 implementing effectively, 171
URL paths, in REST, 112–14 selecting, 177
User experience (UX), 5, 27, 255 server notification using, 171–72
User interface sticky note, 56 WebSocket
User interface (UI) tests, 228–29 bidirectional notification via, 174–76
User sticky note, 56 selecting WebSocket protocol, 178
UX (user experience), 5, 27, 255 Wright, Frank Lloyd, 79
Writing job stories, for APIs, 37–38
when desired outcome is known, 37–38
V when digital capability has been identified, 38
when problem is known, 37
Vernon, Vaughn, 26, 72 WS-Transaction specification, 135–36
Versioning helper libraries, 220–21
Virtual machine (VM), 277
Virtual private network (VPN), 282 X
VM (virtual machine), 277
XML Schema, 244
VOC (voice of the customer), 35
Vogels, Werner, 261
Voice of the customer (VOC), 35
VPN (virtual private network), 282
Y
YAML, 126, 235, 240, 245, 247
“You ain’t gonna need it” (YAGNI) principle, 211
W
W3C, 172, 174
Z
WAFs (Web application firewalls), 277, 278
Web APIs, 3–5, 9–10 Zero trust architecture (ZTA), 282
Photo by izusek/gettyimages
• Automatically receive a coupon for 35% off your next purchase, valid
for 30 days. Look for your code in your InformIT cart or the Manage
Codes section of your account page.
• Download available product updates.
• Access bonus material if available.*
• Check the box to hear from us and receive exclusive offers on new
editions and related products.
*
Registration benefits vary by product. Benefits will be listed on your account page under
Registered Products.
Addison-Wesley • Adobe Press • Cisco Press • Microsoft Press • Pearson IT Certification • Que • Sams • Peachpit Press