A fixed-horizon A/B test asks a static question: split traffic evenly, run until you hit a pre-computed sample size, then read the result. A multi-armed bandit asks a moving one: which arm looks best right now, and how much traffic should it get this hour? Both are legitimate ways to run an experiment, but they optimize for different things, and conflating them is why teams end up with bandit output they try — and fail — to read like a standard A/B result. This guide is about picking the right tool and knowing what each one costs you.

Two Machines, Two Different Bets

A fixed-horizon test is built to answer "which variant is better, and by how much, with a stated error rate?" It holds allocation constant (usually 50/50) so that every unit of traffic contributes equally to a single, clean effect estimate at the end. The cost of a wrong-looking variant during the test is fully absorbed — you don't get to react to it until the test is over.

A multi-armed bandit is built to answer "how do I maximize outcomes while I'm still learning?" It shifts traffic toward whichever arm currently looks best, using algorithms like epsilon-greedy, Thompson sampling, or UCB (upper confidence bound) to balance trying new information against cashing in on what it already believes. The cost of a wrong-looking variant is partially avoided in real time, because the bandit routes fewer users to it as evidence accumulates.

Neither is "more rigorous" in the abstract. A fixed-horizon test is more rigorous for producing a comparable, decision-grade effect size. A bandit is more efficient for minimizing the traffic spent on a bad variant while you're finding the good one.

The Exploration-Exploitation Tradeoff, Concretely

Every bandit algorithm is a running answer to one question: spend this next visitor on exploration (learning more about an arm you're unsure of) or exploitation (routing to the arm you currently believe is best)?

• Epsilon-greedy exploits the current best arm most of the time and explores at random a small fixed fraction of visits (commonly single-digit percentages). Simple, but it keeps exploring at the same rate even after the answer is obvious.
• Thompson sampling draws a plausible effect from each arm's posterior distribution and routes the visitor to whichever arm "wins" that draw. Exploration shrinks naturally as posteriors narrow, without a hand-tuned schedule.
• UCB-based methods route to the arm with the highest plausible upper bound on its true rate, which explicitly favors arms with wide uncertainty — new arms get a boost until they've earned or lost it.

The practical effect across all three: adaptive allocation moves traffic away from underperforming arms before you have enough data to say precisely how much worse they were. That's the point of a bandit, but it's also why bandits are a poor source for the kind of effect-size number you'd want to compare against a StatFacts insight card.

Where Adaptive Allocation Quietly Costs You Statistical Power

The traffic a bandit saves you from a losing arm is traffic it also withholds from measuring that arm precisely. Three consequences follow directly:

• Unequal, time-varying sample sizes. By the end of a bandit run, arms rarely have comparable sample sizes, and the ratio was different at every point in the run. Standard confidence-interval math assumes fixed allocation; applying it to bandit data understates uncertainty.
• Estimates are conditioned on their own history. An arm that looked good early gets more traffic, which makes its later estimate look even more confident — a mild feedback loop that biases naive point estimates optimistically. Specialized adaptive-inference methods correct for this; a plain conversion-rate calculation does not.
• Small or late-breaking effects can get starved out. If a genuinely better arm looks mediocre in its first few hundred visits — plausible for a modest true effect — a well-tuned bandit will already be diverting traffic away from it before enough data accumulates to reverse that impression.

None of this makes bandits wrong. It means the number a bandit reports as "arm B's rate" is a decision-support figure, not a substitute for the kind of effect estimate you'd log against a benchmark.

A Side-by-Side Comparison

| Dimension | Fixed-horizon A/B test | Multi-armed bandit | ||

Related Resources from StatFacts:

• For guidance on interpreting effect ranges, confidence labels, and sample context on any insight card: How to Read Benchmarks Effectively
• To size a fixed-horizon test or estimate the traffic a bandit phase would need: Benchmark Calculator