How to run A/B tests: an experimentation guide for startups
How A/B testing works, forming a hypothesis, choosing a metric, sample size and duration, reading results and common mistakes.
By Editorial Team
An A/B test shows two versions of a page, email or feature to different user groups to measure which performs better. It replaces guesswork with data and is a core tool of growth hacking.
How it works
- Users are randomly split into two groups.
- Group A sees the current version (control), group B the changed one.
- The chosen metric is compared between groups.
- If the difference is too large to be chance, the winner is rolled out.
1. Form a hypothesis
A good hypothesis states the change, the expected result and the reason: "If we highlight the annual plan on the pricing page, more users will choose it, because they currently miss the discount." Hypotheses should come from interviews and data; see our customer interview guide.
2. Choose the primary metric
Pick one primary metric per test: sign-up, payment or click rate. Also define guardrail metrics that must not get worse, such as refunds, churn or page speed.
3. Sample size and duration
- Detecting small differences needs many users. Low-traffic startups should test big, bold changes rather than tweaks.
- Run for a pre-set period, usually at least one or two weeks to cover weekday patterns.
- Do not stop early because results look good; that leads to false conclusions.
4. Read the results
- Check whether the difference is statistically significant.
- Check guardrail metrics before rolling out.
- Record the learning whatever the result; failed tests are valuable too.
What to test
- Headline and value proposition; see our landing page guide
- Number of sign-up form fields
- Pricing page layout; see our pricing guide
- Email subject lines
- Onboarding steps
Common mistakes
- Testing many changes at once in one test
- Hunting for small differences with too little traffic
- Stopping tests early
- Looking only at short-term metrics
Conclusion
A/B testing increases a startup's learning speed. A few well-designed tests beat many scattered ones. To pick the metric that matters, see our North Star metric guide.
This guide is for general information only and is not legal, financial or investment advice. Check official sources and consult professionals for current terms.
