By ByteBot
Cloudflare just drew a line in the sand. Starting September 15, AI training bots and agent bots will be blocked by default on ad-supported pages — across all new sites and every existing free-tier account. That covers the vast majority of sites running behind the world’s largest CDN. If you run a site, you need to understand what changed, what defaults you’re now inheriting, and whether you need to act before the deadline hits.
Cloudflare’s Three-Crawler Taxonomy
To understand the new defaults, you first need the taxonomy Cloudflare created to classify AI traffic. Before this update, “AI crawler” was a catch-all that lumped together fundamentally different behaviors. Cloudflare now separates them into three categories:
- Search — Crawlers that index your content to answer questions later. This includes Googlebot, BingBot, and Applebot. These remain allowed by default.
- Agent — Crawlers used by AI agents that browse the web in real-time to take actions on behalf of users. Blocked by default on ad pages starting September 15.
- Training — Crawlers that consume your content to train or fine-tune AI models. No referral traffic comes back. Blocked by default on ad pages starting September 15.
The taxonomy isn’t just semantic. It determines what your site allows — and the defaults are about to change in a way that affects every developer with a free Cloudflare account. Cloudflare’s full announcement is worth reading: new AI traffic options for all customers.
What Changes on September 15
Here is exactly who gets the new defaults:
- New Cloudflare customers (all plans)
- New sites added by existing customers
- All existing sites on the free plan
For those accounts, Training and Agent crawlers will be blocked by default on pages that serve ads. Search crawlers stay allowed. Paid plan users on existing sites retain their current settings and can configure controls manually through the AI Crawl Control dashboard under Security.
If you’re on a paid plan and haven’t touched AI Crawl Control, you’re not directly affected by the default change — but you’re also leaving the door open. Now is a good time to review what’s coming through.
The Mixed-Use Catch Most Developers Will Miss
Here’s where it gets complicated. Cloudflare applies the most restrictive rule to crawlers that serve multiple purposes. A crawler that does both Search and Training is subject to both rules. Block Training, and that mixed-use crawler is also blocked — even if you wanted its search indexing behavior.
This matters because several major crawlers are classified as mixed-use. If you apply a blanket Training block, you may inadvertently block crawlers you want indexing your content. Cloudflare’s dashboard lets you set per-crawler rules, so you can override the default for specific bots. Read the category labels before setting a blanket block.
The Monetization Gateway: The Opportunity in This Story
Cloudflare’s response to training bots isn’t just blocking — it’s a business model. The Monetization Gateway, currently accepting waitlist signups, lets you charge for any resource behind Cloudflare: a web page, a dataset, an API endpoint, or an MCP tool call.
The underlying protocol is x402, which finally puts HTTP’s long-unused 402 Payment Required status code to work. A crawler requests a resource, your server returns a 402 with a price and payment address, the caller pays in stablecoin, and re-requests with proof of payment. Cloudflare verifies and grants access — sub-second, no payment infrastructure required on your end.
Partners Ceramic.ai and You.com are already integrated. Ceramic pays publishers every time their content appears in its AI search results. You.com pays per access to premium content by AI agents. If you create content with genuine training value, this is a revenue stream worth registering for now.
Why Cloudflare Had to Move
The numbers explain this policy better than any press release. As of June 2026, bots now generate 57.5% of HTML web traffic — the first time automated requests have held the majority over human visitors. Of all AI crawler requests, 51.8% are for Training purposes, up from 22% in Spring 2025. Training crawlers return essentially zero referral traffic. They consume bandwidth, content, and infrastructure — and send nothing back.
Cloudflare’s move is heavy-handed in one sense: a CDN is now making normative decisions about who deserves access to published content. But the alternative — honoring a voluntary robots.txt that training bots routinely ignored — clearly wasn’t working.
What To Do Before September 15
- Check your current settings. In the Cloudflare dashboard, go to Security > AI Crawl Control. Review which crawlers are currently reaching your site and their classifications.
- Decide on your defaults. If you’re on a free plan, Training and Agent bots will be blocked by default after September 15. Confirm this aligns with your goals, or configure per-crawler exceptions for bots you want to allow.
- Join the Monetization Gateway waitlist. If you run a content-heavy site, a technical blog, a dataset, or an API, register before it fills. The waitlist is in your Cloudflare dashboard.
September 15 isn’t a suggestion. Cloudflare is the network layer — it enforces what robots.txt only requests. Decide your position before the default kicks in.
