A/B Testing
“A controlled experiment comparing two versions by randomly showing each to different users.”
Two people on the team have opposite opinions about what will work. Both sound plausible. You've been going back and forth for a week.
A/B testing is a controlled experiment technique. You show two variants to randomly split user groups, measure performance against a pre-defined metric, and let the data decide which version wins.
Its strength is removing opinion from decisions — the numbers don't care who's senior. Reach for it when you have enough traffic to achieve statistical significance and a clear metric worth improving. It excels at optimisation — conversion, engagement, retention — but fails badly for innovation, where both variants might miss the point entirely.
The other trap is post-hoc rationalisation: if you don't define your success metric before the test runs, you'll slice the data until something looks good. And without sufficient volume, you'll read significance into noise.
Your next move: What success metric have you written down before running the next test — and would you actually accept a 'no significant difference' result, or would you keep slicing the data until something looked good?
What it looked like for them
Business Pundit, mid-2000s. The founders wanted to build a paid question-and-answer platform for business topics and ran straight into the two-sided market problem: experts wouldn't sign up without questions, askers wouldn't sign up without experts. Rather than guess which side to seed, they built a cheap blog widget that posted questions into the wild with a 70/30 revenue split to incentivise expert participation. The widget was the test.
It generated real data about how people behaved in the actual context — what kinds of questions got asked, what quality of answers came back, whether the economics worked at the split they'd chosen. The data said no. Complex enterprise questions couldn't be commoditised cleanly enough for the revenue model to sustain itself.
They shut it down. The cost of the experiment was weeks of work and a blog widget. The cost of building the full platform without the experiment would have been months and a team. Testing the proposition cheaply before committing to it is the whole discipline.
“A stakeholder keeps changing their mind and I can't lock the scope.”