Faster A/B tests, same traffic: what AI changes

In short

AI can build A/B tests in minutes, and that is real progress. But orders, not build time, decide how many tests a store can run. The changes that matter most on Shopify, such as checkout, prices, discounts, tax and apps, need more than a fast front-end build. Write a brief for every test, size it before launch, run it in full weeks, check the result against your Shopify orders, and name the person who will analyse it before it starts.

The problem fast building solves

A program that runs one experiment a month has a predictable problem. Every test becomes precious. Teams hunt the backlog for the idea that will definitely work, and meanwhile the new banner and the new offer go live around the program, unmeasured. Across the different markets I have worked in, testing has been essential to understanding the audience. When tests are hard to launch, the most important changes skip them.

AI attacks that problem directly. Testing platforms now let an AI assistant create, launch and read experiments through MCP servers. Convert's MCP server lets you read results, run audits and build tests in plain language. Intelligems' MCP, from a Shopify-focused competitor, can create and update price tests, offers, content and post-purchase upsells. An audit of 14 A/B testing tools published by Convert, itself a testing vendor, found that creating experiments through chat is the most common AI feature, and that 58% of the 59 AI features it tagged were chat interfaces over actions the tools already had (page updated November 3, 2026). In other words, most of the AI so far speeds up building.

The speed helps most when it comes with habits that have nothing to do with AI:

A documented, falsifiable hypothesis for every idea, somewhere the whole business can see it.
Ideas scoped across the whole site, not only the page they came from, each with a rollout plan if it wins.
A request form open to everyone, so testing becomes the fastest route to a change instead of the slowest.

Building was never the only bottleneck

When one stage of a process gets faster, the bottleneck moves to the next one. I wrote about this for software delivery in Finding delivery bottlenecks: AI made writing code faster, so the work piled up at review. Experiments behave the same way. Once building a test takes minutes, the slow parts become:

Traffic. Every test needs enough visitors and orders to tell a real effect from noise, and tests compete for the same visitors.
Definition. A clear hypothesis, scope and metric, decided before launch.
Analysis. Someone who reads the result honestly, checks it against the real data, and decides.

In other companies, I have seen what happens when a team gets a testing tool and only the first part speeds up: many tests, no defined scope, a test stopped after a couple of days, and nobody responsible for analysing the data. That produces activity, not learning.

Traffic decides how many tests you can run

Before launching a test, you can calculate how many visitors it needs. The inputs are your current conversion rate, the smallest lift worth detecting, and how sure you want to be. Here is an example with a 2% conversion rate, 95% confidence (two-sided) and 80% power, using the standard formula for comparing two conversion rates. The traffic figure is illustrative: a page that gets 5,000 visitors a day, split between control and variant.

Lift you want to detect	Conversion rate	Visitors per variant	Orders in total (both variants)	Days at 5,000 visitors a day
5%	2.00% → 2.10%	about 315,000	about 12,900	about 126 (18 weeks)
10%	2.00% → 2.20%	about 80,700	about 3,400	about 32 (5 weeks)
20%	2.00% → 2.40%	about 21,100	about 930	about 8 (2 weeks)

Three things follow from this:

Small lifts need a lot of traffic. Halving the lift you want to detect roughly quadruples the visitors you need. Revenue per visitor, a common headline metric, usually needs more still, because order values vary much more than a yes-or-no conversion.
Run in full weeks. Weekend shoppers behave differently from weekday ones. Round every duration up to whole weeks, even when the calculation says eight days.
Parallel tests share a fixed budget. Two tests on the same visitors either split them, and each takes twice as long, or overlap, and can affect each other. Tests built in a morning still have to fit into the traffic you have.

Vendors frame this differently, which is a reason to run the numbers yourself. On its podcast episode on traffic (June 15, 2026), Intelligems suggests that around 500 to 700 orders a month is likely enough for meaningful Shopify tests, and that the number that matters is orders, not visitors. The same episode makes a point worth keeping: average order value matters. Its example is a furniture brand selling $4,000 to $5,000 couches that might process only 80 to 100 orders a month, too few to reach significance in useful time, and its advice for lower traffic is to take bigger swings. Set against the table, that threshold only works for large changes. The example page above already produces about 3,000 orders a month. At 600 orders a month, detecting a 10% lift needs about 3,400 orders, or five to six months; a 20% lift needs about 930 orders, or six to seven weeks. So 500 to 700 orders a month is enough if you are looking for lifts of about 20% or more, which is exactly why lower-volume stores should test bigger changes.

This is a calculation an AI assistant can run in seconds. Make it part of every brief, so the team knows before launch whether a test can answer its own question.

Win rate is the wrong score

A Convert case study shows the pattern. A leisure and tourism client's previous agency ran about one test a month: 12 tests in a year at a 33% win rate, which Convert works out as about four winners. In its first three months, the agency Another Web Is Possible ran 16 tests, built with its own tooling connected to Convert's MCP server, and found four winners, a 25% win rate. The best one raised revenue per visitor by 12%.

Read one way, the new program wins less often. Read properly, it found the same number of winners in a quarter of the time. Learning per month is the right score, and on that the faster program comes out ahead. But winners are only a rough measure of learning: as the next point shows, a program that runs more tests also collects more false winners, so the same count of winners is not quite the same amount of learning. Inconclusive and losing tests teach something too, if they are recorded, and the case study makes the same point about its losing tests.

Running 16 tests in three months takes plenty of traffic, tests spread across different parts of the site, tests sized for large lifts, or a mix of the three. The case study doesn't give traffic or test durations, so I can't say which. On a store, tests on the home page, collections, product pages and cart can each use most of the traffic, but shoppers move between those pages, so the tests still overlap and can affect each other, a risk to accept knowingly. Faster building makes parallel tests practical; it doesn't create the traffic for them.

Two cautions come with running more tests, though:

More tests, more false winners. With a two-sided test at 95% confidence, each test of a change that does nothing has about a 2.5% chance of looking like a winner. If none of 16 changes had any real effect, the chance that at least one would still show up as a winner is about one in three. Count significant losers too and it rises to more than one in two. That doesn't make any of the case study's winners false. It means a bigger program needs more checking: re-test important winners, or keep a small group of visitors on the old version after rollout.
The best result is usually an overestimate. When you pick the biggest lift out of many tests, chance has usually helped it. This is known as the winner's curse. Plan the rollout around a smaller lift than the test showed, and measure again after it ships.

So for every test, record how long it ran, how it shared traffic with other tests, and whether the winner held after rollout. Case studies rarely include these, and this one doesn't, but your own program should. Without them, a list of winners is hard to learn from, however fast it was produced.

What a fast build doesn't reach on Shopify

Our stores run on Shopify, and tax, apps and other third-party services all connect through it. AI that writes variant code is excellent at the front-end layer: layout, copy, images, banners, product page changes. The changes that usually matter most to revenue sit deeper:

Checkout. Checkout is no longer a place to paste scripts. Shopify no longer supports checkout.liquid on the information, shipping and payment steps, and script tags on the thank-you and order status pages were sunset for non-Plus stores on August 26, 2026. To change checkout, apps now use extensions, including web pixel extensions for tracking. Convert, for example, states that because of these changes, direct checkout testing is currently unavailable in its Shopify app.
Prices, discounts and shipping thresholds. These change what the customer pays, the amount tax is calculated on, and what every app that reads the cart sees. If you sell in more than one country or currency, the same test plays out differently in each market.
Apps. Reviews, subscriptions, bundles, upsells and tax apps add their own code to the page or read the cart. A variant that moves an element an app depends on can break it in that variant only. The test then reports a loss that is really a bug, or misses the problem entirely.
Speed. Most client-side testing tools hide the page until the variant is applied, or visitors see the original flash first. Either way the tool itself has a cost, which I measured on real stores in Check before you believe.

Deeper tests are possible, but they are integration work, not a morning's build. Intelligems, which specialises in Shopify price tests, says in its integration guide that it is moving price tests from Checkout Scripts, which Shopify is sunsetting, to Shopify Cart Transform Functions. The guide lists four steps: add its script, tag product prices in the theme, update the cart, then QA and publish. It also notes that only one Cart Transform Function can run on a single line item, and asks stores that already use one for something else to contact its support first. That is exactly the kind of dependency an engineering review should catch before a price test starts.

Shopify now has a native option too. Its Rollouts feature can schedule changes to the theme, checkout and accounts pages, discounts and catalogs, and its Experiment type compares a treatment against a control. Shopify's requirements page lists limits worth knowing:

Experiments need the Grow plan or higher.
Theme, checkout and accounts changes apply only to the online store. Headless and custom storefront checkouts can't be tested, and analytics exclude headless and POS visitors.
Rollouts don't work on vintage themes and can't change Liquid templates.
In a discount Experiment, the control group gets no rollout discount, and combination settings can make different discounts eligible in each group.
When a catalog change is in an Experiment, or a rollout reaches less than 100% of visitors, prices and products can differ between the online store and Google, other channels, ads, feeds and third-party apps.

The feature is new and still changing, so check Shopify's help center for the current details.

None of this makes front-end tests less useful. It means every test needs a technical scope: is it front-end only, or does it touch checkout, prices, tax, apps or markets? The second kind needs engineering review, QA in every market, and a rollback plan, however fast the variant was built.

Revenue per visitor needs one source of truth

A testing tool usually records revenue when the purchase happens. Your Shopify orders hold the real number: after discounts, without tax and shipping, and net of the refunds and cancellations that arrive weeks later. If the two disagree, the orders are right. It's the same reason I pushed for a single source of truth when we built our data platform.

Define the metric in Shopify's terms before launch. For example: net sales excluding tax and shipping, after refunds within 30 days, divided by visitors.
Connect each order to its variant. For example, write the experiment and variant to the cart as a cart attribute, so it lands on the order and you can reconcile the tool's numbers with your own. Check with your testing vendor how it records this, since each tool does it differently.
Check the split before reading the result. If you set 50/50 and got a clearly uneven split on thousands of visitors, something is broken: bots, redirects, caching, consent or a variant that fails to load. Fix that first. The result can't be trusted until you do.
Decide the outlier rule in advance. A handful of very large orders can swing revenue per visitor. Cap order values at a set percentile, or exclude wholesale orders, and decide it before the test starts, never after seeing which variant the big order landed in.

The test tool can't see your calendar

Across markets, I have learned to look well beyond the page being tested, because the result depends on everything else happening at the same time. A testing tool sees visitors and clicks. It doesn't see:

Campaigns and strategy. An email blast, a paid campaign or a promotion changes who arrives and why. Look at the full picture of what marketing is running before and during the test.
Expected traffic for that window. A test sized on last month's traffic may run short in a slow month, or be swamped by a sale.
Country, currency and season. Exchange rates, local holidays and the time of year all change how people buy.
Weather. For seasonal products, a warm weekend can move sales more than any variant.
Other changes to the site. If someone edits the tested page or launches a new banner mid-test, the test is no longer measuring one thing. Freeze the tested area, or at least log every change.

Simulations don't fix this either. Shopify's SimGym runs AI shoppers through a theme before launch, and it can be a cheap filter for weak ideas. But Shopify itself notes that results might differ from how actual customers behave, and simulated shoppers have no season, budget or campaign behind their visit. Use simulation to decide what to test, and real traffic to decide what worked.

So keep a shared calendar with every test on it next to campaigns and promotions. And keep the history: every test with its hypothesis, dates, result and what else was happening. Months later, with more data, you can go back, check the result against external data, and analyse it again.

Before launch: a one-page brief

Hypothesis. What we change, for whom, what we expect to happen, and the evidence for it.
Metric and guardrails. One primary metric defined in Shopify's terms, plus a few that must not get worse, such as average order value, refunds and page speed.
Size and duration. The smallest lift worth detecting, the visitors and orders needed, and the end date in full weeks.
Stopping rule. The test ends on its end date. Checking daily and stopping as soon as it looks significant makes false winners far more likely, unless the tool uses a sequential method designed for it. Stop early only for a real problem, such as a broken variant.
Technical scope. Front-end only, or does it touch checkout, prices, tax, apps or markets? If it goes deeper, engineering reviews it and a rollback plan is ready.
QA. Every market, currency and device, with every app on the page working in both versions.
Calendar check. Campaigns, promotions, holidays and season during the test window.
Owner. The person who will analyse the result, and the date they will present it.
After the result. The rollout plan if it wins, how you will confirm the lift after shipping, and the entry in the test log.

Where AI belongs in the program

The answer is not less AI. It is AI on the slow parts too, with people keeping the decisions. Some tools are starting to move that way. Optimizely's Experiment Review Agent can review an experiment's configuration before launch and recommend changes to improve its odds of reaching statistical significance. Intelligems says its MCP skills build what they recommend as a draft for you to review, and Convert's local MCP server defaults to a read-only reporting profile. Check how approvals work in whichever tool you use before letting an assistant touch a live store.

Stage	Where AI helps	What stays with the team
Brief	Drafts the hypothesis from research and past tests	Whether the idea deserves the traffic
Sizing	Calculates visitors, orders and duration	The smallest lift that matters to the business
Build and QA	Writes the variant, screenshots every device and market	Whether it touches checkout, prices, tax or apps
Monitoring	Checks the test's health every day (split, errors, speed), not its results	Whether a problem is real enough to stop
Analysis	Reconciles results with Shopify orders and applies the outlier rule	What the result means, and whether to roll it out
Log	Writes the entry and links related tests	What we actually learned

The same kind of MCP connection that lets AI build a test can let it read orders, so it can do the reconciliation as well as the build. I covered how to make those connections useful in Building MCP servers that help non-technical users.

AI took the build off the critical path, and that is worth celebrating. But a testing tool with AI improves speed only up to a point. What turns speed into learning is being strategic and analytical: a clear definition, a scope the platform can support, enough traffic, honest analysis and data you have checked. Speed without definition just produces wrong conclusions faster.

Sources

Sample sizes and false-positive odds were calculated with the standard two-proportion formula (95% confidence two-sided, 80% power).

Erwin MonguiDirector of Technology at Cedar Planters and Certified AI Engineer, based in Toronto. More about me