A/B Testing
One-Line Definition
Compare the real impact of two options through controlled experiments.
Core Concept
Helps you verify key assumptions with minimal cost, avoiding big investments based on gut feelings.
More specifically, A/B testing is suited to answer questions like: Is what I’m seeing a fact, a hypothesis, or a habitual practice? To make a better choice, which variable, path, or constraint should be examined first?
When to Use
- When the problem becomes complex and intuition is no longer reliable enough.
- When the team disagrees on the next move and needs a shared analysis framework.
- When you need to convert abstract judgments into concrete actions, checklists, or experiments.
- When current practices are losing effectiveness and you need to re-examine the underlying logic.
When NOT to Use
- The problem is simple, and direct execution matters more than analysis.
- There is a lack of basic facts, leading to concept spinning with no grounding.
- The model is used only to justify an existing conclusion rather than to help correct the judgment.
- The cost of testing is extremely high with no room for trial and error, and no additional means of verification exist.
How to Apply
- Write down the current problem: Describe in one sentence what you need to judge or solve.
- List existing assumptions: Distinguish between facts, opinions, experience, emotions, and default answers given by others.
- Identify key variables: Find the 1–3 factors that most influence the outcome.
- Formulate actionable options: Propose a few different approaches based on the key variables.
- Define a minimal validation: Use a low-cost action to verify which judgment is closer to reality.
Example
Imagine a team discovers that the conversion rate for new users is dropping. When applying “A/B Testing,” instead of immediately asking the designer to change a button or having operations increase the budget, they first break it down: Where are the users coming from? What information do they see? At which step do they hesitate? What do they lose when they leave? Is there a stronger alternative available? After breaking it down, the team may find that the real problem is not insufficient traffic, but that users do not understand what problem the product solves on the very first screen. Therefore, the minimal action is not to rebuild the entire product, but to first test a clearer value proposition.