An Onboarding A/B Test Result That Surprised the Whole Team

The hypothesis going in was unremarkable: reducing onboarding from seven steps to three would reduce drop-off and increase activation. It's the standard CRO playbook — fewer steps, less friction, more completions. We ran the test expecting a clean, quick win. The seven-step version beat the three-step version on 30-day retention by a meaningful margin.

Why fewer steps lost

The three-step version did have a higher completion rate — more people finished onboarding, exactly as predicted. But fewer of them stuck around. The longer flow asked users to configure specific preferences (which data sources to connect, which team members to invite) that the short flow skipped by using sensible defaults. Users who'd personally configured those settings during the longer onboarding treated the product as something they'd set up, not something that had been handed to them pre-filled. The extra friction was also extra investment, and that investment predicted retention better than the completion rate did.

Reducing friction increases how many people finish onboarding. It doesn't automatically increase how many people stick around, and those are not the same success metric.

What this changed about how we test onboarding now

• Measure 30-day retention as the primary success metric for onboarding tests, not completion rate, which only captures the first few minutes.
• Distinguish friction that's pure cost (confusing copy, broken steps) from friction that's actual configuration investment — only the first kind should be aggressively cut.
• Test removing individual steps rather than collapsing the whole flow at once, to isolate which specific steps were driving the retention effect.
• Stop treating 'shorter onboarding' as a default best practice and treat it as a hypothesis that needs the same rigor as any other test.

We ended up with a middle version — five steps, keeping the two configuration steps that drove the retention effect and cutting the three that were pure friction with no corresponding investment value. It beat both the three-step and seven-step versions, but only because the test forced us to understand why the long version worked instead of just assuming shorter was always better.