Stefan Thomke

Though they’ve been warned for decades about the dangers of overrelying on gut instinct and personal experience, managers keep failing to critically examine—much less challenge—the ideas their decisions are based on. To correct this problem they need to think and act like scientists. That requires doing five things: (1) being a knowledgeable skeptic and relentlessly questioning assumptions; (2) investigating anomalies—things that are unexpected or don’t look right; (3) devising testable… Show more

Though they’ve been warned for decades about the dangers of overrelying on gut instinct and personal experience, managers keep failing to critically examine—much less challenge—the ideas their decisions are based on. To correct this problem they need to think and act like scientists. That requires doing five things: (1) being a knowledgeable skeptic and relentlessly questioning assumptions; (2) investigating anomalies—things that are unexpected or don’t look right; (3) devising testable hypotheses that can be quantifiably confirmed or disproved; (4) running experiments that produce hard evidence; and (5) probing cause and effect.

Drawing on the experiences of Harrah’s Entertainment, Sony, Bank of America, and Lego, Thomke and Loveman show how scientific methods can help companies discard ineffective practices, increase marketing and operational efficiency, boost customer satisfaction and sales, find new sources of growth, and even turn around struggling businesses. Show less

Why don’t organizations test more? After examining this question for several years, I can tell you that the central reason is culture. As companies try to scale up their experimentation capacity, they often find that the obstacles are not tools and technology but shared behaviors, beliefs, and values. For every online experiment that succeeds, nearly 10 don’t—and in the eyes of many organizations that emphasize efficiency, predictability, and “winning,” those failures are wasteful. To… Show more

Why don’t organizations test more? After examining this question for several years, I can tell you that the central reason is culture. As companies try to scale up their experimentation capacity, they often find that the obstacles are not tools and technology but shared behaviors, beliefs, and values. For every online experiment that succeeds, nearly 10 don’t—and in the eyes of many organizations that emphasize efficiency, predictability, and “winning,” those failures are wasteful. To successfully innovate, companies need to make experimentation an integral part of everyday life—even when budgets are tight. That means creating an environment where employees’ curiosity is nurtured, data trumps opinion, anyone (not just people in R&D) can conduct or commission a test, all experiments are done ethically, and managers embrace a new model of leadership. In this article, I look at several companies that have managed to do those things well. Show less

When it comes to improving customer experiences, trying out new business models, or developing new products, even the most experienced managers often get it wrong. They discover that intuition, experience, and big data alone don't work. What does? Running disciplined business experiments. And what if companies roll out new products or introduce new customer experiences without running these experiments? They fly blind.

That's what Harvard Business School professor Stefan Thomke shows in… Show more

When it comes to improving customer experiences, trying out new business models, or developing new products, even the most experienced managers often get it wrong. They discover that intuition, experience, and big data alone don't work. What does? Running disciplined business experiments. And what if companies roll out new products or introduce new customer experiences without running these experiments? They fly blind.

That's what Harvard Business School professor Stefan Thomke shows in this rigorously researched and eye-opening book. It guides you through best practices in business experimentation, illustrates how these practices work at leading companies, and answers some fundamental questions: What makes a good experiment? How do you test in online and brick-and-mortar businesses? In B2B and B2C? How do you build an experimentation culture? Also, best practice means running many experiments. Indeed, some hugely successful companies, such as Amazon, Booking.com, and Microsoft, run tens of thousands of controlled experiments annually, engaging millions of users. Thomke shows us how these and many other organizations prove that experimentation provides significant competitive advantage.

How can managers create this capability at their own companies? Essential is developing an experimentation organization that prizes the science of testing and puts the discipline of experimentation at the center of its innovation process. While it once took companies years to develop the tools for such large-scale experiments, advances in technology have put these tools at the fingertips of almost any business professional. By combining the power of software and the rigor of controlled experiments, today's managers can make better decisions, create magical customer experiences, and generate big financial returns.

Experimentation Works is your guidebook to a truly new way of thinking and innovating. Show less

The field of customer experience (CX) design — which aims to ensure that customers have positive touch points with companies while buying and consuming their products and services — has grown quickly in recent years. Research has shown that memorable experiences, and the ensuing positive word of mouth, can drive customer decisions as much as, if not more than, price and functionality. To that end, consultants have created thoughtful tools and frameworks such as journey mapping, service… Show more

The field of customer experience (CX) design — which aims to ensure that customers have positive touch points with companies while buying and consuming their products and services — has grown quickly in recent years. Research has shown that memorable experiences, and the ensuing positive word of mouth, can drive customer decisions as much as, if not more than, price and functionality. To that end, consultants have created thoughtful tools and frameworks such as journey mapping, service blueprinting, and problem-solving mindsets. Academics have studied customer engagement models that focus on managerial variables such as employee selection, training, rewards, and service culture. Yet recent research reports suggest that there have been few, if any, meaningful improvements in customer experience over time. Despite the insights gleaned about customers through advanced technologies and data analysis, something still seems to be missing for most companies.

My experience and research points to the missing ingredient: emotion. The most memorable experiences are suffused with emotion — not extra features or value for money. Show less

In the fast-moving digital world, even experts have a hard time assessing new ideas. Case in point: At Bing a small headline change an employee proposed was deemed a low priority and shelved for months until one engineer decided to do a quick online controlled experiment—an A/B test—to try it out. The test showed that the change increased revenue by an astonishing 12%. It ended up being the best revenue-generating idea Bing ever had, worth $100 million.

That experience illustrates why… Show more

In the fast-moving digital world, even experts have a hard time assessing new ideas. Case in point: At Bing a small headline change an employee proposed was deemed a low priority and shelved for months until one engineer decided to do a quick online controlled experiment—an A/B test—to try it out. The test showed that the change increased revenue by an astonishing 12%. It ended up being the best revenue-generating idea Bing ever had, worth $100 million.

That experience illustrates why it’s critical to adopt an “experiment with everything” approach, say Kohavi, the head of the Analysis & Experimentation team at Microsoft, and Thomke, an HBS professor. In this article they describe how to properly design and execute A/B and other controlled tests, ensure their integrity, interpret results, and avoid pitfalls. They argue that if a company sets up the right infrastructure and software, it will be able to evaluate ideas not only for improving websites but also for new business models, products, strategies, and marketing campaigns—all relatively inexpensively. This will help it find the right path forward, especially when answers aren’t obvious or people have conflicting opinions. Show less

The data you already have can't tell you how customers will react to innovations. To discover if a truly novel concept will succeed, you must subject it to a rigorous experiment. In most companies, tests do not adhere to scientific and statistical principles. As a result, managers often end up interpreting statistical noise as causation--and making bad decisions. To conduct experiments that are worth the expense and effort, companies need to ask themselves several questions: (1) Does the… Show more

The data you already have can't tell you how customers will react to innovations. To discover if a truly novel concept will succeed, you must subject it to a rigorous experiment. In most companies, tests do not adhere to scientific and statistical principles. As a result, managers often end up interpreting statistical noise as causation--and making bad decisions. To conduct experiments that are worth the expense and effort, companies need to ask themselves several questions: (1) Does the experiment have a clear purpose? Managers must figure out exactly what they want to learn in order to determine if testing is the best approach. (2) Have stakeholders made a commitment to abide by the results? Are they willing to walk away from a project if the findings suggest they should? (3) Is the experiment doable? The complexity of the variables in a business experiment and their interactions can make it difficult to determine cause-and-effect relationships. Choosing the right sample size is important. (4) How can we ensure reliable results? Randomized field trials, "blind" tests, and big data can help. (5) Have we gotten the most value out of the experiment? Conducting the experiment is just the beginning. Use the data to assess which components of a new initiative might have the highest ROI or the markets where it is most likely to be successful. Show less

Why do certain product and service experiences seem to have that undeniable “wow” factor, while others disappoint customers? Perhaps there’s no better place to turn to than the world of magic. Stefan Thomke and Jason Randal consider that leading magicians are constantly under pressure to come up with new “effects” that wow audiences. They have to innovate frequently and rely on a systematic way of doing so.

Think you need exceptional employees, advanced IT, or rigid controls to build a high-performance organization? The dabbawalas of Mumbai prove otherwise. Six days a week, these 5,000 self-managed, semiliterate workers deliver upwards of 130,000 lunches from customers' homes to their offices with astonishing precision--negotiating the crowded city by train, bicycle, and handcart, without the aid of any technology or even cell phones. The 100-year-old service is legendary for its reliability:… Show more

Think you need exceptional employees, advanced IT, or rigid controls to build a high-performance organization? The dabbawalas of Mumbai prove otherwise. Six days a week, these 5,000 self-managed, semiliterate workers deliver upwards of 130,000 lunches from customers' homes to their offices with astonishing precision--negotiating the crowded city by train, bicycle, and handcart, without the aid of any technology or even cell phones. The 100-year-old service is legendary for its reliability: Despite monsoons, floods, riots, and terrorist attacks, mistakes by the dabbawalas are extremely rare. Thomke, an HBS professor, studied the dabbawalas to find the keys to their success. Show less

Many companies approach product development as if it were manufacturing, trying to control costs and improve quality by applying zero-defect, efficiency-focused techniques. While this tactic can boost the performance of factories, it generally backfires with product development. The process of designing products is profoundly different from the process of making them, and the failure of executives to appreciate the differences leads to several fallacies that actually hurt product-development… Show more

Many companies approach product development as if it were manufacturing, trying to control costs and improve quality by applying zero-defect, efficiency-focused techniques. While this tactic can boost the performance of factories, it generally backfires with product development. The process of designing products is profoundly different from the process of making them, and the failure of executives to appreciate the differences leads to several fallacies that actually hurt product-development efforts. In this article, the authors, an HBS professor and a consultant, expose these misperceptions and others. Show less

Managing Product and Service Development is about the managerial aspects critical to conceiving, designing, and developing innovative products and services. The course exposes students to some of the best management practices, tools, and frameworks known today, and introduces new approaches that hold promise for the future. Many texts are either aimed at engineering or marketing specialists and do not adequately address the often difficult general management issues that arise in complex… Show more

Managing Product and Service Development is about the managerial aspects critical to conceiving, designing, and developing innovative products and services. The course exposes students to some of the best management practices, tools, and frameworks known today, and introduces new approaches that hold promise for the future. Many texts are either aimed at engineering or marketing specialists and do not adequately address the often difficult general management issues that arise in complex development project. This book does not require training or experience in a technical field but addresses the role of new technologies in product development.

In this text students learn about innovation through exploration. All the material has been developed and tested in the MBA and executive education classroom at Harvard Business School. The Instructor’s Manual (IM), as only part of this text’s proven teaching materials, describes an optional student project that complements in-class sessions. Show less

Every company's ability to innovate depends on a process of experimentation whereby new products and services are created and existing ones improved. But the cost of experimentation often limits innovation. New technologies—including computer modeling and simulation—promise to lift that constraint by changing the economics of experimentation. Never before has it been so economically feasible to ask "what-if" questions and generate preliminary answers. These technologies amplify the impact of… Show more

Every company's ability to innovate depends on a process of experimentation whereby new products and services are created and existing ones improved. But the cost of experimentation often limits innovation. New technologies—including computer modeling and simulation—promise to lift that constraint by changing the economics of experimentation. Never before has it been so economically feasible to ask "what-if" questions and generate preliminary answers. These technologies amplify the impact of learning, paving the way for higher R&D performance and innovation and new ways of creating value for customers.In Experimentation Matters, Stefan Thomke argues that to unlock such potential, companies must not only understand the power of experimentation and new technologies, but also change their processes, organization, and management of innovation. He explains why experimentation is so critical to innovation, underscores the impact of new technologies, and outlines what managers must do to integrate them successfully. Drawing on a decade of research in multiple industries as diverse as automotive, semiconductors, pharmaceuticals, chemicals, and banking, Thomke provides striking illustrations of how companies drive strategy and value creation by accommodating their organizations to new experimentation technologies.As in the outcome of any effective experiment, Thomke also reveals where that has not happened, and explains why. Probing and thoughtful, Experimentation Matters will influence how both executives and academics think about experimentation in general and innovation processes in particular. Experimentation has always been the engine of innovation, and Thomke reveals how it works today. Show less