A single A/B test can be run with a point tool and a spreadsheet. An organization running dozens of concurrent tests across web, mobile, email, and backend logic needs an experimentation platform: shared infrastructure for consistently bucketing users into variants, preventing tests from silently interfering with each other, centralizing statistical analysis, and giving every team the same trustworthy source of truth for 'did this ship a real improvement.' This is the architecture behind internal systems at companies like Netflix, Airbnb, and Spotify, and behind commercial platforms like Statsig, Eppo, and GrowthBook.

The core components

A mature experimentation platform generally has four architectural layers:

Assignment service: deterministically buckets a given user/session into a variant for a given experiment, usually via consistent hashing of a user ID + experiment ID, so the same user always lands in the same variant for that experiment without needing to store a lookup table for every user.

Exposure logging: records every time a user is actually exposed to a variant (not just eligible for it) — a critical distinction, because analyzing conversion rates against everyone *eligible* rather than everyone actually *exposed* silently dilutes real effects.

Metrics pipeline: joins exposure events to downstream conversion/revenue events (often in a warehouse like BigQuery or Snowflake) to compute each variant's metric values.

Analysis/stats engine: runs the significance or Bayesian computation and surfaces results, ideally with automated checks like Sample Ratio Mismatch detection built in.

exposure_to_conversion_join.sql

sql

-- Classic experimentation metrics pipeline pattern: join exposure events
-- to the relevant downstream conversion event, scoped to users who were
-- actually exposed to a variant (not just eligible for the experiment).
WITH exposures AS (
  SELECT
    user_id,
    experiment_id,
    variant,
    MIN(exposure_timestamp) AS first_exposure_ts
  FROM analytics.experiment_exposures
  WHERE experiment_id = 'pricing_page_headline_v3'
  GROUP BY user_id, experiment_id, variant
),
conversions AS (
  SELECT user_id, MIN(event_timestamp) AS conversion_ts
  FROM analytics.purchase_events
  GROUP BY user_id
)
SELECT
  e.variant,
  COUNT(DISTINCT e.user_id) AS exposed_users,
  COUNT(DISTINCT c.user_id) AS converted_users,
  COUNT(DISTINCT c.user_id) / COUNT(DISTINCT e.user_id) AS conversion_rate
FROM exposures e
LEFT JOIN conversions c
  ON e.user_id = c.user_id
  AND c.conversion_ts > e.first_exposure_ts  -- only count conversions AFTER exposure
GROUP BY e.variant

Counting eligible users instead of exposed users dilutes every result

A frequent, severe bug: analyzing conversion rate against everyone who *could* have seen a variant (e.g. everyone who hit the eligible page) rather than everyone who was *actually* exposed to the rendered variant (accounting for ad blockers, slow-loading JS, cache hits that served stale content). This denominator mismatch systematically dilutes real effects toward zero, making genuinely winning tests look like non-significant ties. Exposure logging has to be tied to the actual client-observed rendering event, not the server-side eligibility decision.

Experiment interference and mutual exclusion

Running many concurrent experiments on overlapping populations risks interference: a pricing test and a checkout-flow redesign test running simultaneously on the same users can each distort the other's results in ways that are hard to detect after the fact. Mature platforms solve this with layers/namespaces (Google's and Microsoft's internal systems popularized this pattern) — each user is assigned to exactly one experiment within a given layer, guaranteeing mutual exclusion for experiments that might interact, while allowing truly independent experiments in different layers to run on overlapping traffic simultaneously.

Build vs. buy for experimentation infrastructure

Approach	Example	Fit
Point testing tool only	Optimizely Web, VWO	A handful of web-only tests, low concurrency
Feature-flag-based platform	Statsig, Eppo, GrowthBook, LaunchDarkly	Many concurrent experiments across web, backend, and mobile
Homegrown on top of warehouse	Custom assignment + dbt/BigQuery metrics	High engineering maturity, full control, significant build cost

The platform's real value is trust, not features

The actual payoff of investing in experimentation platform architecture isn't running more tests faster — it's that when a result says a variant won, every team trusts the number enough to act on it without re-litigating the methodology each time. That trust comes specifically from consistent exposure logging, automated SRM checks, and a single shared stats engine, not from any individual test's UI.

What's Next?

The assignment and bucketing mechanism at the heart of an experimentation platform is architecturally identical to a feature flag — which is exactly why many teams build or adopt both on the same underlying infrastructure.

I build these systems professionally.

Whether it's a RAG pipeline, analytics migration, or AI workflow — let's talk.

Work With Me →

Need production AI or MarTech expertise? I build custom LLM workflows, server-side GTM tagging, and analytics infrastructures professionally.

Need custom AI or MarTech setup? Let's build together.

Let's talk →