Sample size math for multivariate tests

01The baseline arithmetic

Every sample size number in this post comes from one formula. For a single planned comparison between two proportions, the required number of users per arm is

n ≈ (z₁₋₂ + z₁₋)² × [p₀(1−p₀) + p₁(1−p₁)] / (p₁ − p₀)²

where p₀ is baseline conversion, p₁ is the rate you’re trying to detect, z₁₋₂ is the critical value for your significance level (1.96 for two-sided α=0.05), and z₁₋ is the critical value for your target power (0.84 for 80% power). The two terms in brackets are just the Bernoulli variance of each arm — a proportion’s variance is p(1−p), and it’s largest around p=0.5, which is why low-baseline conversion tests actually need less variance-driven sample size per unit of absolute lift than people expect, but need much larger relative lifts to produce a meaningful absolute one.

Run the numbers for a plain 2-arm test at a 3% baseline, targeting a lift to 3.5% (a +0.5 percentage-point absolute lift, roughly +17% relative), 80% power, two-sided α=0.05:

n ≈ (1.96+0.84)² × [0.03×0.97 + 0.035×0.965] / (0.005)²
n ≈ 7.849 × 0.0630 / 0.000025 ≈ 19,780 users per arm

That’s roughly 39,560 total users for a single A/B comparison at a modest, realistic conversion rate and a modest lift. Nothing about this is exotic — it’s the textbook two-proportion z-test sample size formula, the same one behind every online significance calculator. The point of showing it is that it doesn’t get gentler when you add more arms; it gets much less gentle, for reasons the rest of this post works through with real numbers.

02What a factorial design multiplies

A factorial design crosses every level of every factor into cells — a 2x2 (two factors, two levels each) has 4 cells; a 3x3 has 9. The sample size formula above still applies, but now it applies per contrast, not per cell, and which contrast you care about changes the answer by an order of magnitude.

Label the four cells of a 2x2 design 00, 01, 10, 11 (factor A off/on, factor B off/on). If there’s no interaction, the main effect of A is just the difference between the average of the two A=1 cells and the average of the two A=0 cells — each side of the comparison pools two cells together, which is exactly the two-proportion problem from section 1 applied to the marginal rates. Concretely: baseline 2%, main effect lift to 2.5% (+0.5pp), 80% power, α=0.05, gives 6,904 users per cell, or 27,616 total across the four cells — almost identical total traffic to the plain 2-arm test in section 1, because the pooling exactly offsets having four cells instead of two.

The interaction is a different, much more expensive animal: it’s the difference-in-differences, Δ₳ = p₁₁ − p₁₀ − p₀₁ + p₀₀, and it doesn’t get the pooling benefit — each cell contributes its full individual variance with no averaging-down. Suppose both factors independently lift conversion from 2% by +0.5pp, and you want to detect an extra +0.5pp of synergy on top of that additive baseline: that needs 32,063 users per cell, or 128,252 total — about 4.6x the traffic of the main-effect test at the same baseline and same-sized effect. This is the single most under-appreciated fact in MVT planning: a factorial design gets main effects almost for free, but interactions are never free, and teams that only budget for main-effect traffic are systematically underpowered to say anything real about interactions.

A 3x3 design (one ordered factor, three levels, e.g. a discount of 0%/5%/10%) adds a second choice: test the omnibus 2-degree-of-freedom main effect, or, if you believe the effect is monotonic, a preplanned 1-degree-of-freedom linear trend. At a 5% baseline where each step up adds +0.5pp (5.0% → 5.5% → 6.0%), the omnibus test needs 3,329 users per cell (29,961 total); the linear trend test needs only 2,719 per cell (24,471 total) — about 18% less traffic for asking a more specific, and usually more realistic, question.

None of this is optional once you have more than one contrast in play. Testing A, B, and A×B at the nominal 5% significance level each means roughly a 14% chance that at least one is a false positive purely by chance, before you’ve looked at a single number. The standard fix is a Bonferroni correction — test each hypothesis at α/m instead of α (m=3 tests here means α=0.0167 each) — or its slightly less conservative cousin Holm’s step-down procedure. Both tighten the critical value, which raises the required n: at 80% power, correcting a 3-hypothesis family this way inflates one-degree-of-freedom sample sizes by about 1.334x, and an 8-component family (e.g. testing linear and quadratic contrasts across a 3x3 grid) by about 1.629x. Skip the correction and you haven’t saved that traffic — you’ve just spent it manufacturing winners that won’t replicate.

03Fractional factorials

A fractional factorial runs only a fraction — a half, a quarter — of the full cross-product of cells, using a carefully chosen subset so the design stays balanced and orthogonal. The catch is aliasing: because you didn’t run every cell, some effects become mathematically indistinguishable from others. Design resolution tells you which: Resolution III means main effects are tangled up with two-factor interactions; Resolution IV keeps main effects clean of two-factor interactions but tangles those interactions with each other; Resolution V keeps both main effects and two-factor interactions clean of one another (they only alias with three-factor interactions, which are usually negligible). Higher resolution costs more runs for the same factor count.

Here’s the part that surprises most digital teams: fractionating a design barely changes the total traffic needed for a target main-effect precision. In a balanced 2^(k−p) fraction, each factor level still pools over half the remaining cells, so per-cell sample size roughly doubles every time you halve the number of cells — and the two effects cancel. Two worked screening examples show this concretely:

Total traffic barely moves. What a fraction actually buys you is fewer variants to design, build, and QA, and it costs you interaction visibility — a Resolution III quarter-fraction of four two-level factors aliases every main effect with some two-factor interaction, so you can no longer tell whether the effect you measured is factor A or the A×C interaction wearing A’s name. Choose a fraction because you’ve decided which interactions you can live without estimating cleanly — not because someone told you it “saves traffic.” With only two factors, fractionating is close to useless: a half-fraction of a 2x2 collapses to 2 cells and makes the interaction inestimable outright. Fractional designs earn their keep once you’re screening four or more factors at once.

04Practical stop rules

Every number above assumes a fixed-horizon design: you compute n in advance, run the test until each arm hits that count, and analyze once. That assumption is load-bearing, and it’s the same discipline this blog has argued for elsewhere: you should never edit a running experiment’s parameters mid-flight, and the same logic applies to peeking at results. “Run it two weeks and see” is not a stop rule — it’s an invitation to peek at the results daily and stop the moment a difference looks significant, which inflates your real false-positive rate far above the nominal 5% you designed for. Checking a test five times and stopping at the first p<0.05 you see pushes your actual Type I error rate well past 20%, not 5%.

If you genuinely want to look at the data before the planned end — for a kill switch on an obviously bad variant, say — use a group-sequential design with a formal α-spending function, which allocates your total error budget across the planned looks instead of letting each look use a full 5%. Pocock boundaries use roughly the same, moderately strict critical value at every look. O’Brien-Fleming boundaries are much stricter early and relax toward something close to the standard fixed-horizon threshold at the final look, which is usually the better default for teams that mostly want a bailout option, not an early-stopping habit. Either way, the exact sample-size and traffic impact depends on how many looks you plan and when — decide that before launch, not while watching the dashboard.

05A calculator you can steal

Put the master formula from section 1 in a spreadsheet with four inputs — baseline p₀, target p₁, α, power — and one output column per contrast you plan to test. If you’re running a factorial with more than one hypothesis, multiply the single-contrast n by the correction factor for your chosen multiplicity method before you commit traffic. A few reference points, all at two-sided α=0.05 and 80% power:

*Per-arm figure for the single 2-arm comparison (not a per-cell factorial figure); see section 1.

Read it as a checklist, in order: (1) state your MDE in absolute percentage points, not relative percent — a “10% lift” at a 1% baseline is +0.1pp, at a 10% baseline it’s +1.0pp, and those are not remotely the same experiment; (2) decide whether you actually need interaction estimates, and if so, budget the 4-6x multiplier, not the main-effect number; (3) if any factor is ordered, plan the linear-trend contrast instead of the omnibus test; (4) pick your multiplicity correction — Holm/Bonferroni for a strict launch decision, Dunnett if the real question is “which cell beats the control,” Benjamini-Hochberg only if this is an exploratory screen you intend to confirm later; (5) fix the stop rule before you route a single user. Do all five and the number you land on is one you can defend, not one you backed into after the test was already running.