Cloudflare Blocks Dual-Purpose AI Crawlers on September 15 and Publishers Have to Pick a Side

For two years the argument about AI scraping has been mostly rhetorical. Publishers complained, lawyers filed, everyone waited. On September 15 it stops being rhetorical, because Cloudflare starts blocking multi-purpose crawlers by default unless the site owner opts out.

Multi-purpose is the load-bearing word. A crawler that only indexes for search has never been the problem. A crawler that only harvests training data was easy to ban in robots.txt and easy to shame in public. The bots that broke the model are the ones doing both jobs on the same pass, so that the same fetch which earns you a place in the results also feeds the system that answers the query without sending anyone to your page.

That dual role is why the old defence stopped working. You could always turn the bots off. Turning them off also removed you from search. Publishers have been staring at that trade for a year and blinking, because losing your ranking to protect content you were already losing is a difficult memo to write for the board.

What actually changes

Cloudflare sits in front of a very large share of the web, which means the default matters more than the policy. Most site owners never touch a default. If the switch flips to blocked and stays blocked across a few million properties, the crawl budgets of every company building answer engines take a real cut overnight, not because anyone won an argument but because a CDN changed a setting.

Cloudflare’s own numbers put bot traffic above half of everything moving across the web. That figure gets quoted as a curiosity. It’s actually the business case. If more than half your bandwidth serves machines, and the machines increasingly return nothing, the hosting bill is a subsidy you’re paying to your competitor.

The framing from Cloudflare’s side has been consistent: publishers should be findable without having to hand the content over for free. Whether that’s achievable is a technical question nobody has answered yet. Discoverability and extraction use the same pipe. Splitting them requires either the AI companies to identify themselves honestly or the infrastructure layer to guess, and guessing is how you end up blocking Googlebot on a Tuesday and losing a quarter of your traffic.

The part publishers are not ready for

Opting out will be tempting for a lot of sites, and the ones that opt out fastest will be the ones with the least leverage. A small publisher looking at flat referrals and a shrinking ad rate does not feel like refusing free exposure, even exposure that pays badly. The big groups can afford to test the block because they have direct audiences, apps, newsletters and licensing deals to fall back on.

So the likely outcome is not a united front. It’s a split, with the top of the market gating and negotiating while the long tail stays open and gets scraped harder because there’s less competition for it. That’s roughly what happened with paywalls, and roughly what’s happening now with search referrals.

There’s a second problem underneath. Blocking a crawler protects your content from that crawler. It does nothing about the copy of your article on an aggregator, the summary in a newsletter, the rewrite on a content farm, or the version already inside a model trained two years ago. The block is prospective. The damage is not.

Watch the negotiation, not the block

Treat September 15 as the opening of a negotiation rather than a wall going up. The interesting question is what gets agreed in the six months after it. Per-crawl pricing has been floated. So have collective licensing pools, pay-per-inference arrangements, and preferred-partner deals where a handful of large publishers get paid and everyone else gets the standard terms.

Any of those would be a genuine change in how the web is monetised, the first since programmatic advertising. It would also, very likely, be a change that rewards scale. The publishers who can name a price are the ones whose absence would be noticed.

The rest find out what their content is worth when it becomes optional.