What Is A/B Testing?
Updated 17 September 2026
Jump to section
A/B testing, also called split testing, compares two versions of a web page against each other to see which one performs better. You show version A to half of your live visitors and version B to the other half, then measure which earns more conversions. The winner counts only when the result clears a confidence threshold, called statistical significance, usually set near 95%.
A/B testing replaces opinion with evidence. Instead of arguing over which design is better, you let real visitors decide.
How does an A/B test work?
An A/B test runs both versions at the same time, splitting live traffic between them so outside factors hit each version equally. Optimizely, a testing platform, defines it as comparing two versions of a page to determine which performs better.
The steps are consistent:
- Form a hypothesis, such as "a shorter form will lift inquiries."
- Build version B, changing one thing only.
- Split traffic evenly between A and B.
- Run it until you have enough data, then keep the winner.
| Part of a test | What it is |
|---|---|
| Version A | The current page, kept as the control |
| Version B | The variant, changed in one way |
| Hypothesis | The single change you expect to help |
| Winner | The version that comes out ahead on real data |
Changing one thing at a time is what makes the result readable. If you change the headline and the button and the image at once, a win tells you nothing about which change caused it.
What is statistical significance?
Statistical significance is the confidence that the difference between your two versions is real, and not random noise. Optimizely describes it as the likelihood that the gap in conversion rates is not due to chance.
This is why small tests mislead. If 10 people see each version, and one buys on B while none buy on A, that looks like a win. But it sits well within the range of luck (illustrative example, not a client result).
Most testing tools calculate significance for you, and wait until they are about 95% confident before calling a winner. The mechanism matters: without that threshold, you ship changes based on coincidence and wonder why results never improve.
What can you test?
You can test any element that might change a visitor's decision on a landing page, from the words to the layout. The highest-value tests usually sit closest to the action.
Common things to test include:
- The headline and the main promise
- The call-to-action wording and color
- Form length and the fields you ask for
- Page layout and the order of sections
- Images, and how the price is shown
In our experience, testing a few options beats committing to one design on instinct, so we favor a real test over a confident guess. A test earns its place when it changes something a visitor actually weighs, like the offer or the call to action, not the shade of a button. A/B testing is one tool inside conversion rate optimization, and a heatmap often points to the element worth testing first.
What tools run A/B tests?
Several tools run A/B tests, though the best-known free option has closed. Google Optimize, Google's free testing tool, stopped working on 30 September 2023.
Google now points users to third-party tools that connect to Google Analytics, where you track conversions, naming AB Tasty, Optimizely, and VWO. Some website platforms and email tools also include their own split-testing features.
For a small Malaysian business, the tool matters less than the traffic. Testing needs a steady flow of visitors to reach significance, so a brand-new site often gains more from fixing obvious problems than from formal tests.
Frequently asked questions
How much traffic do I need for A/B testing?
Enough to reach a trustworthy result, which usually means hundreds of conversions per version, not just visits. A page with a handful of inquiries a month cannot run a fair test, because the numbers are too small to separate a real effect from luck. Low-traffic sites do better fixing clear friction first, like a slow page, then testing once volume grows.
Is A/B testing the same as multivariate testing?
No. An A/B test compares two whole versions that differ by one change, while a multivariate test varies several elements at once to find the best combination. Multivariate testing needs far more traffic, because it splits visitors across many variations. Most small businesses should stay with simple A/B tests, which give clear answers on realistic traffic.
How long should an A/B test run?
Long enough to reach statistical significance and to cover a full business cycle, usually at least one to two weeks. Stopping early, the moment one version pulls ahead, is a common mistake, because the lead often vanishes as more data arrives. Run the test across weekdays and a weekend, since visitor behavior changes by day.
Can A/B testing hurt my SEO?
No, when done normally. The risk is cloaking, which means deliberately showing search engines different content from people. Keep both versions honest, avoid hiding content from Google alone, and if variations sit on separate URLs, use a canonical tag so Google knows which one to index. Testing for real visitors is safe.
What happened to Google Optimize?
Google Optimize, its free A/B testing tool, closed on 30 September 2023, along with Optimize 360. Google retired it and now recommends third-party platforms that integrate with Google Analytics instead. Existing users had to move to other tools. If an old guide tells you to use Optimize, it is out of date.
Testing your way to more sales
Storming Solutions builds and maintains websites for Malaysian businesses, and we would rather test a change than argue about it in a meeting. When a page matters, we look at how people actually use it before we touch the design.
Not sure what to test first on your site? Send us the page on WhatsApp, or start with the basics in our guide to conversion rate optimization, then talk to us about web development.