A/B Test Hypothesis for Meta Ads: Template and Test Cost

An A/B test hypothesis is a prediction that one specific change to an ad will move one metric by at least a stated amount, written so the data can contradict it. You show control (A) and variant (B) to comparable audiences and see whether the data supports the prediction.

Statisticians pair the prediction with two formal statements:

• Null hypothesis (H0): the variant and control have the same underlying rate. Any gap you see is noise.
• Alternative hypothesis (H1): the variant's rate differs from control's. You size the test to detect a difference of at least your minimum lift.

The test asks whether the data is surprising enough under H0 to reject it. Nothing is proven true. You either have evidence against "no difference" or you do not.

Five terms recur on this page:

• Baseline: control's current rate for the metric.
• Minimum lift: the smallest improvement worth acting on, as a percentage of baseline. A baseline of 3.0% and a minimum lift of 20% means a target of 3.6%.
• Primary metric: the single number that decides the test.
• Guardrail: a second number that must not get worse, such as cost per purchase.
• Margin of error: the range around your measured difference within which the true difference probably sits, at your chosen confidence level.

A hypothesis card holds nine fields: the change, the audience and placement, the primary metric with baseline and target, the guardrail, the reason, the null and alternative hypotheses, the cost to test, the look date, and the decision for a win, a flat result and a loss. Fill it in before you build the ad. If you cannot fill a field, the test is not ready.

HYPOTHESIS CARD
Change:        [one variable] from [control] to [variant]
Audience/placement: [who, where]
Primary metric: [metric], baseline [x%], target [x% x (1 + minimum lift)]
Guardrail:     [metric that must not worsen]
Because:       [evidence: customer review, comment, competitor ad, past test]
H0 / H1:       No difference / differs by at least [minimum lift]
Cost to test:  [clicks per arm] x [cost per click] x 2 = $____
Look date:     the later of [date at least 7 days out] and [clicks per arm] reached.
               No decision before it.
If win / flat / loss: [ship and log] / [decision] / [kill and log reason]

A filled example, with illustrative numbers you should replace:

Change: first three seconds from a product shot to a customer-review quote. Audience/placement: prospecting audience, Feed. Primary metric: click-to-purchase rate, baseline 3.0%, target 3.6% (a 20% lift). Guardrail: cost per purchase. Because: review text repeatedly mentions fit, which the product shot never shows. Cost to test: 13,914 clicks per version x $1.20 x 2 = about $33,400. Look date: the first day on or after day 7 on which both versions have 13,914 clicks. If flat: keep the product shot and test a bolder review angle.

The "because" line is where most weak hypotheses fail. A reason drawn from your own evidence, such as reviews, support tickets or what competitors run repeatedly, gives the test something to learn from even when it loses.

Run the test in Meta's A/B test tool in Ads Manager, which splits the audience so the two versions do not overlap. Meta's Business Help Center page on A/B tests advises running a test for at least about 7 days, so a large budget that reaches the click count in a day or two still waits for day 7, which also smooths day-of-week swings.

Cost scales with the inverse square of the lift: halving the improvement you want to detect roughly quadruples the clicks needed. The tables below show this for a store with a 3% click-to-purchase rate.

Our model. For a two-sided test at 95% confidence and 80% power, the clicks needed per version are:

n = (1.9600 × √(2 × p̄ × (1 − p̄)) + 0.8416 × √(p1 × (1 − p1) + p2 × (1 − p2)))² ÷ (p2 − p1)²

Here p1 is the control rate, p2 the variant rate and p̄ their average. 95% confidence means you accept a 5% chance of calling a win when nothing changed. 80% power means an 80% chance of catching a real lift of the size you named. Inputs: a 3% baseline, a $1.20 cost per click (so about $40 per purchase), and two versions. These are example inputs, not market benchmarks. We worked these figures out ourselves in November 2026.

Going from a +20% to a +10% target takes clicks from 13,914 to 53,210, about 3.8 times as many. The cheapest row costs about one twenty-first of the dearest ($6,041 against $127,705, before rounding).

As a cross-check, the rule of thumb of 16 × p × (1 − p) ÷ (difference)² in Evan Miller's sample-size guidance gives about 51,700 clicks per version at +10%, within 3% of the table's 53,210. At +20% it gives about 12,900, within 8% of 13,914.

The model assumes clicks are independent and your purchase rate stays steady during the test. Real accounts drift with seasons and promotions, so treat each row as a floor. Fix one conversion window before launch so baseline and result are counted the same way: see our page on the Meta ads attribution window.

To run this with your own baseline and cost per click, use our Meta ads budget and creative test calculator.

Two moves cut the bill without making the hypothesis vaguer: test a bigger change, or move the primary metric up the funnel. A third check keeps the test from stalling. None of them removes the need to state a minimum lift.

Test a bolder change. A +50% target needs about 2,500 clicks per version in the table above. A new angle can plausibly move results that far. A tweak to button text usually cannot.

Use a cheaper metric deliberately. Click-through rate (CTR, the share of impressions that become clicks) moves faster than purchases. We modeled a 1% baseline CTR at $12 per 1,000 impressions (CPM, cost per mille), using the same formula, 95% confidence and 80% power. A $12 CPM at a 1% CTR equals the $1.20 cost per click used above.

A CTR win is a cheaper test, but it is a proxy. In our view, use it to screen hooks and angles, then confirm the survivors on purchases or cost per purchase before you scale. Your own CTR baseline matters more than any benchmark; our page on what a good CTR for Facebook ads is shows how to find it.

Check that your budget can reach the click count. If a test is too thinly funded to reach its clicks within a sensible window, shrink the scope instead of watching it drag on. Our page on the Meta ads learning phase covers whether a given budget will exit it.

Stopping a test the first time one version is ahead raises the false-positive rate well above the 5% you chose. The significance calculation assumes you decided the sample size in advance and looked once.

Checking after every new observation at a 5% threshold produces a false positive 26.1% of the time when there is no real difference, and peeking ten times turns what looks like 1% significance into about 5%, according to Evan Miller's analysis of repeated significance testing (April 18, 2010). His fix is to fix the sample size in advance and not test repeatedly.

We also simulated it in November 2026: 1,500 tests per setting, each with a true 3% rate in both versions and 4,000 users per version. At each look, spaced evenly through the test, we ran a two-sided two-proportion z-test at the 5% level, and counted a test as a false winner if any look was significant. Run-to-run noise (one standard error) is about 0.5 points at 4.5% and about 1 point at 20.1%.

Our figures are lower than Miller's 26.1% because we looked at 5 or 10 points, not after every observation. The direction is the same. The practical rule is the look date on your card: write down the click count or date, and make no decision until you reach it. Our guide to creative test stop rules covers when to end a test.

Decide before launch what each outcome triggers, because a flat result is the most common outcome and the one teams handle worst. A flat result means the data cannot separate the variant from control at the size you powered for. It does not mean the change does nothing.

Our recommended decision table, written onto the card:

Logging matters as much as deciding. The next hypothesis should start from what the last ones showed, and a flat test with its reason written down is still evidence. Our Brand Brain, the memory layer in the platform, keeps a record of what worked, what failed and why, so a failed concept is not rerun by accident. Our own guidance is to state the expected outcome and the decision criterion before launch, as in our note on creative testing memory.

If you want to see how a ranked weekly test plan could sit on top of your account, ask for pricing and a walkthrough when you book a personalized demo with our founding team.