Migrating A/B Testing Platforms Mid-Experiment, Lessons Learned

A platform contract ended, a new one had to go live, and three experiments were already running. We planned the migration carefully. Two experiments survived with their data intact. The third had to be thrown out entirely, and the reason why became the core lesson of the whole migration.

What we got right

For two of the experiments, we ran both platforms in parallel for a week before cutting over, confirmed bucketing consistency against a shared random seed logic, and migrated the running experiment's variant assignments so returning users kept seeing the same variant they'd already been bucketed into. Continuity of individual user experience was the thing we protected hardest, and it paid off — those two experiments' results held up against pre-migration trend lines with no visible discontinuity.

A platform migration mid-experiment isn't really a tooling problem. It's a user-continuity problem wearing a tooling costume.

Where the third experiment died

The third experiment used the old platform's native audience-targeting logic to exclude a specific user segment, logic the new platform implemented slightly differently at the edges — a timing difference in how recently-active users got classified. That small difference meant some users who should have been excluded were newly included after the migration, contaminating the sample in a way we couldn't retroactively correct. We killed the experiment and restarted it clean on the new platform rather than trying to salvage mixed data.

• Any experiment relying on platform-specific targeting logic, not just raw bucketing, needs to be audited line-by-line against the new platform's equivalent feature before migration.
• Parallel-running for a week caught the first two issues before they became data problems; it would have caught the third too if we'd included targeting logic in the parallel-run check, not just assignment consistency.
• We now treat 'does this experiment depend on platform-specific targeting' as the first migration-risk question, before bucketing or stats methodology.
• Restarting a contaminated experiment clean cost two weeks of lost data, but it was cheaper than the credibility cost of reporting a result we weren't confident in.

The migration plan's biggest blind spot wasn't the platforms' statistics engines, which were similar enough. It was the much less glamorous targeting and audience logic sitting underneath the experiment, which is exactly the part migration checklists tend to skip.