Cloudflare just launched a crawl endpoint that lets you scrape entire websites with a single API call. Sounds helpful, right? Here’s the catch: Cloudflare controls 20% of global web traffic and aggressively blocks scrapers with bot protection. Now they’re selling you access to the same content they block. That’s not innovation. That’s extortion.
What Cloudflare Built
The /crawl endpoint, launched March 10 in open beta, automatically discovers and renders web pages through Cloudflare’s Browser Rendering service. Submit a URL, get a job ID, poll for results in HTML, Markdown, or JSON. It respects robots.txt, handles JavaScript rendering, and costs $0.09 per browser hour plus $2 per concurrent browser beyond the free tier.
Cloudflare markets this for AI training, RAG pipelines, and content monitoring. On paper, it’s a legitimate tool.
The Conflict of Interest
Here’s the problem: Cloudflare profits from both sides of web scraping.
On one side, they sell Bot Management to websites. This service blocks “unrecognized bot traffic” by default. Their documentation admits that “even if your intent is harmless, your scraper may still get locked out.” Researchers, journalists, and developers trying to access public data get blocked constantly.
On the other side, Cloudflare now sells scraping access. The /crawl endpoint works because Cloudflare doesn’t block itself. Pay the fee, and suddenly those “malicious” requests become “legitimate” API calls to the exact same content.
A Hacker News commenter captured it perfectly: “It’s hard to see how this isn’t extorting folks by offering a working solution that, oh, cloudflare doesn’t block.”
Cloudflare created the problem, then monetized the solution. That’s the definition of a racket.
Monopoly Power Enables This
This only works because Cloudflare has gatekeeping power. They don’t just run a scraping service — they control access to half the web.
The numbers: Cloudflare serves 20% of global internet traffic. They protect 48.7% of the top one million websites. Among sites using CDNs, 79.9% choose Cloudflare.
If your target site runs behind Cloudflare, the /crawl endpoint is the only guaranteed-to-work solution. You can try rotating proxies, headless browsers, or bypass tools like Scrapling, but Cloudflare’s bot detection catches most of them. Or you can pay Cloudflare for access they’re actively preventing.
This isn’t a competitive market. It’s a tollbooth on a highway Cloudflare built.
Who Gets Hurt
AI companies with billion-dollar budgets can afford Cloudflare’s fees. Researchers, journalists, and independent developers can’t.
The free tier offers 10 minutes of scraping per day. That’s useless for anything beyond testing. The paid tier includes 10 hours per month, then charges $0.09 per hour and $2 per concurrent browser. At scale, costs explode.
Web scraping isn’t just for tech companies. Journalists scrape court records and government databases to investigate corruption. Researchers scrape social platforms to study misinformation. Activists scrape corporate data to expose labor violations. These use cases serve the public interest, but they don’t come with venture funding.
Even more absurd: Cloudflare profits from content they don’t own. Website owners don’t see a cent of the scraping fees. Cloudflare charges for access to someone else’s data, pockets the money, and calls it innovation.
This creates a two-tier web. If you can pay, you get access. If you can’t, you’re locked out of public data by a private gatekeeper.
The Counterargument
Cloudflare will say they’re protecting websites from malicious scrapers. Fair point. DDoS attacks, credential stuffing, and aggressive crawlers that crash servers are real problems.
But legitimate research isn’t an attack. Data journalism isn’t malicious. Monitoring price changes or tracking product availability isn’t abuse.
Cloudflare could improve bot classification to distinguish public interest scraping from malicious activity. They could offer free or reduced-cost access for verified researchers and journalists. They could let website owners opt into revenue-sharing for scraping fees.
Instead, they chose the most profitable path: block everyone by default, then sell access back to them.
The Verdict
Cloudflare’s /crawl endpoint isn’t a helpful service. It’s a protection racket.
They control both the lock and the key. They block legitimate access to public data, then charge for the privilege of getting it back. Their market position — 20% of web traffic, half the top million sites — gives them the power to pull this off without competition.
If this succeeds, others will follow. Akamai and AWS CloudFront control another 40% of web access. Three companies could paywall 60% of the web, not by creating content, but by gatekeeping infrastructure.
The open web doesn’t die from attacks. It dies from monopolies monetizing access to content they don’t own.
Developers, researchers, and regulators need to push back before this becomes the norm.
