Cloudflare is fundamentally shifting the landscape of web scraping by introducing a strict policy that forces AI companies to distinguish between traditional search crawlers and those used for AI model training. Beginning September 15, 2026, the company will implement default settings that block ‘mixed-use’ crawlers from accessing any web pages that host advertisements. This change will apply to all new Cloudflare customers, new sites added by existing users, and the entire base of free-tier accounts.

The initiative aims to address the growing tension between website owners and AI developers. While many publishers want their content to remain discoverable via search engines, they are increasingly resistant to having their intellectual property harvested for AI training without compensation or consent. By forcing a separation of these bot functions, Cloudflare intends to give publishers more granular control over how their data is utilized in the age of generative AI.

Beyond blocking unauthorized access, Cloudflare is expanding its ‘Pay Per Crawl’ marketplace into a ‘Pay Per Use’ model. This evolution allows publishers to monetize their content when it provides tangible value to AI services, rather than simply charging for the act of scraping. Initial partnerships with Ceramic.ai and You.com demonstrate this framework, where publishers are compensated when their content is featured in AI-driven search results or accessed by premium AI services. Cloudflare CEO Matthew Prince emphasized that with non-human traffic now exceeding human traffic online, these measures are essential to fostering a sustainable digital ecosystem.

Key Takeaways

• Starting September 15, 2026, Cloudflare will block 'mixed-use' crawlers by default on ad-supported sites to prevent unauthorized AI training.
• The company is evolving its 'Pay Per Crawl' tool into a 'Pay Per Use' model, allowing publishers to monetize content accessed by AI agents.
• Cloudflare aims to force AI companies to separate search-only bots from those that scrape data for model training and agentic services.

Editor’s Analysis & Impact

Cloudflare’s policy shift represents a significant escalation in the ongoing battle over data sovereignty in the AI era. By leveraging its position as a massive web infrastructure provider, Cloudflare is effectively setting a new industry standard that prioritizes publisher rights over the ‘wild west’ scraping practices currently employed by many AI firms. This move forces AI developers to choose between transparency and total exclusion from a vast portion of the internet. If successful, this could lead to a new economic model where high-quality data is treated as a paid commodity rather than a free resource. However, it also risks creating a fragmented web where only sites that comply with specific technical standards remain accessible to AI, potentially deepening the divide between major tech platforms and independent content creators.

Frequently Asked Questions

Q: What happens on September 15, 2026?
A: Cloudflare will update its default settings to block 'mixed-use' crawlers—bots that perform both search and AI training functions—from accessing ad-supported websites.

Q: How does the 'Pay Per Use' model work?
A: It allows publishers to charge AI companies for the value their content provides, such as when it appears in AI-generated search results or is accessed by an AI agent, rather than just charging for the initial crawl.