Interaction effect

When one factor's effect depends on which variant of another factor is present: exactly what A/B testing structurally can't see.

An interaction effect is when the effect of one factor depends on which variant of another factor is present. A new headline might do nothing on its own and a new CTA might do nothing on its own, while the combination of both lifts conversion sharply; or the reverse happens, where two changes that each help independently cancel each other out when shipped together.

A sequence of isolated A/B tests cannot see this by construction: each test only ever varies one factor while holding the others fixed, so it never observes what happens when both move at once. Only a full factorial design (testing every combination at once) can estimate an interaction term directly.