Cloudflare's September 15 AI Crawler Block Turns the Ad Tag Into an Access Key

Thu, 03 Sep 2026 00:00:00 +0000

The most consequential change to how AI systems reach news content this year will not arrive from a courtroom, a licensing negotiation, or a regulator. It arrives from a settings default at a content delivery network, on September 15.

The rule itself is narrow. Cloudflare will change what happens by default to crawlers that blend three jobs into one user agent: indexing pages for traditional search, fetching them for AI agents, and pulling them for model training. On any page that carries advertising, those blended crawlers get blocked unless the site owner decides otherwise. The new default applies to new Cloudflare customers, to new sites set up by existing customers, and to everyone on the free plan. A crawler that will not declare which job it is doing on a given request gets blocked outright on ad-supported pages.

Cloudflare Blocks Dual-Purpose AI Crawlers on September 15 and Publishers Have to Pick a Side

Wed, 26 Aug 2026 00:00:00 +0000

For two years the argument about AI scraping has been mostly rhetorical. Publishers complained, lawyers filed, everyone waited. On September 15 it stops being rhetorical, because Cloudflare starts blocking multi-purpose crawlers by default unless the site owner opts out.

Multi-purpose is the load-bearing word. A crawler that only indexes for search has never been the problem. A crawler that only harvests training data was easy to ban in robots.txt and easy to shame in public. The bots that broke the model are the ones doing both jobs on the same pass, so that the same fetch which earns you a place in the results also feeds the system that answers the query without sending anyone to your page.

Cloudflare on Media Forum

Cloudflare's September 15 AI Crawler Block Turns the Ad Tag Into an Access Key

Cloudflare Blocks Dual-Purpose AI Crawlers on September 15 and Publishers Have to Pick a Side