Sean Ellis

“I found that asking people if they were satisfied with a product didn’t deliver meaningful insights; disappointment was a much better gauge of product loyalty than satisfaction).”
― Sean Ellis, Hacking Growth: How Today's Fastest-Growing Companies Drive Breakout Success

“Let’s say experiment A is testing a small change, such as the color of the sign-up button. As results start coming in, it becomes clear that the increase in the number of new visitors signing up is very small—garnering just 5 percent more sign-ups than the original button color. Besides the obvious assumption that changing the color of the sign-up button may not be the key factor holding back new users from signing up, it’s also an indication that you’ll have to let the experiment run quite a long time in order to have enough data to make a solid conclusion. As you can see from the chart above, to reach statistically significant results for this test, you’d need a whopping 72,300 visitors per variant—or, in other words, you’d have to wait 72 days to get conclusive results. As Johns put it in an interview with First Round Review, “That’s a lifetime when you’re a start-up!” In a case like this what a start-up really ought to do is abandon the experiment quickly and move on to a next, potentially higher-impact, one.”
― Sean Ellis, Hacking Growth: How Today's Fastest-Growing Companies Drive Breakout Success