We Decided to Let AI Crawlers In, Here's the Reasoning

A client's leadership wanted to block every AI crawler in robots.txt as a blanket policy, on the general principle that AI companies shouldn't get free content to train on. I understood the instinct. I also thought it was the wrong call for this specific business, and we spent a real meeting walking through why.

This client's revenue depends heavily on discovery traffic, and a growing share of discovery is happening through AI answer engines citing and summarizing web content directly. Blocking those crawlers doesn't just deny them training data — it removes the possibility of being cited at all in a channel that's clearly growing, in exchange for a principle that wouldn't meaningfully affect how AI models get trained anyway, given how much of the web remains open.

The actual tradeoff we weighed

• Blocking preserves a sense of control and avoids the discomfort of 'giving away' content for free
• Allowing access keeps the door open to AI-driven citations and referral traffic as that channel matures
• The content itself was never behind a paywall or exclusive — it was already fully public and indexable by traditional search
• Selective blocking (certain crawlers, certain content sections) was available as a middle path we hadn't initially considered

Blocking everything on principle feels like control. It's actually just opting out of a channel before you know whether it matters.

Where we landed

We allowed the major AI crawlers broadly but blocked access to a specific set of gated, revenue-adjacent content that genuinely shouldn't be freely summarized elsewhere. It's not a universal answer — a publisher whose entire business model is pageviews on exclusive reporting has a very different calculation than this client did. The point isn't that allowing access is always right, it's that 'block everything on principle' skips the actual business tradeoff in favor of a feeling, and that's worth noticing before you commit to it.