Classical (frequentist) A/B testing asks: 'if there were truly no difference between variants, how unlikely would data this extreme be?' — the p-value. Bayesian A/B testing asks a more directly useful question: 'given the data observed, what is the probability that variant B actually beats variant A, and by how much?' The Bayesian answer is a full probability distribution over the possible true effect size, not a single pass/fail threshold — which maps much more naturally onto the business decision marketers actually need to make ('should we ship B, and how confident are we').

Priors: the part that makes Bayesian methods controversial

A Bayesian model starts with a prior distribution — a belief about the likely conversion rate before seeing any test data — and updates it into a posterior distribution as data accumulates. A weak, uninformative prior (e.g. 'conversion rate could plausibly be anywhere from 0% to 100%, uniformly') lets the data dominate quickly and produces results similar to a frequentist analysis. A strong, informative prior (e.g. 'based on 50 past tests on this page, conversion rates cluster tightly around 4-6%') can meaningfully speed up reaching a confident conclusion, but it also means two analysts running the same test with different priors can reach different conclusions from identical data — which is the core objection critics raise about letting subjective priors influence results. In practice, most marketing Bayesian tools default to weak, mildly informative priors specifically to avoid this criticism.

beta-posterior-update.ts

typescript

// Beta-Binomial conjugate model: the standard Bayesian approach for
// conversion rate testing. Beta(alpha, beta) prior updates cleanly
// into a Beta posterior as conversions/visitors accumulate.
function updateBetaPosterior(
  priorAlpha: number, // e.g. 1 = uninformative prior
  priorBeta: number,  // e.g. 1 = uninformative prior
  conversions: number,
  visitors: number
): { alpha: number; beta: number } {
  return {
    alpha: priorAlpha + conversions,
    beta: priorBeta + (visitors - conversions),
  };
}

// Variant A: 420 conversions / 10,000 visitors
const posteriorA = updateBetaPosterior(1, 1, 420, 10000);
// Variant B: 460 conversions / 10,000 visitors
const posteriorB = updateBetaPosterior(1, 1, 460, 10000);

// P(B > A) is then estimated via Monte Carlo: draw thousands of samples
// from each posterior distribution and count how often B's draw exceeds A's.
function probabilityBBeatsA(a: typeof posteriorA, b: typeof posteriorB, draws = 100000) {
  let bWins = 0;
  for (let i = 0; i < draws; i++) {
    const sampleA = sampleBeta(a.alpha, a.beta);
    const sampleB = sampleBeta(b.alpha, b.beta);
    if (sampleB > sampleA) bWins++;
  }
  return bWins / draws;
}

declare function sampleBeta(alpha: number, beta: number): number;

Frequentist vs. Bayesian A/B testing

Dimension	Frequentist	Bayesian
Core output	p-value, reject/fail-to-reject null hypothesis	Probability distribution; P(B beats A), expected loss
Peeking at interim results	Invalid without correction (inflates false positive rate)	Valid — posterior updates correctly with any amount of data seen so far
Interpretability for stakeholders	Often misunderstood ('95% confident' is commonly misread)	More intuitive ('87% probability B is better')
Sensitivity to subjective input	None — purely data-driven	Sensitive to prior choice, though weak priors minimize this
Tooling maturity in martech platforms	Standard, default in most A/B testing tools	Growing (VWO, some Optimizely features) but less universal

The real selling point: no p-hacking penalty for checking results early

Frequentist significance testing is only statistically valid if you decide on a sample size in advance and look once, at the end — repeatedly checking a dashboard and stopping as soon as p < 0.05 appears inflates the actual false-positive rate far above the nominal 5%. A properly implemented Bayesian posterior can be legitimately examined at any point during the test without this penalty, since it's simply expressing current belief given data seen so far — which matches how marketing teams actually behave (checking test dashboards daily) far better than frequentist assumptions do.

Bayesian methods don't eliminate the need for pre-planned stopping rules

Even though Bayesian posteriors are valid to check at any time, 'I'll stop as soon as P(B beats A) crosses 95%' without any floor on minimum sample size can still stop a test too early on a small, lucky sample before the posterior has had enough data to narrow meaningfully. Pair Bayesian analysis with a minimum sample size or minimum test duration, and consider the width of the posterior credible interval, not just the point probability, before deciding.

What's Next?

Bayesian methods address the 'peeking' problem from one angle; the frequentist world has its own dedicated toolkit for the same problem, built specifically to allow early stopping without inflating error rates — worth understanding even if Bayesian is your primary approach.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →