Cloudflare, which protects around 20% of all websites globally, is changing how AI companies interact with the open web. The company will now block AI crawlers by default on all newly onboarded websites, forcing AI firms to get explicit consent before accessing site content for training or search.
The move is Cloudflare’s latest effort to prevent content scraping, which involves automatically extracting data from websites using AI bots without permission and usually for malicious intent.
This change comes alongside the launch of a new product called Pay per Crawl, a marketplace that allows website owners and media publishers to charge AI bots for scraping their content.
Publishers can set custom pricing depending on the use case – whether the data will be used to train large language models, power real-time search results, or assist independent bots. The product aims to create more transparency and fairness in how AI firms obtain data.
Leading media outlets such as Condé Nast, TIME, and The Atlantic have already joined the program, blaming declining referral traffic on AI-generated answers increasingly reducing the need for users to click through to original sources.
These publishers, among others, argue that without fair compensation, AI scraping undermines their ability to sustain journalism and content creation.
Cloudflare also released comparative crawl-to-crawl ratios that highlight the imbalance between traffic generated and data consumed. OpenAI’s crawlers reportedly make 1,700 requests per referral, while Anthropic’s reach a staggering 73,000-to-1. In contrast, Google’s crawl-to-click ratio sits at a more balanced 14-1.
Story continues below this ad
The move redefines Cloudflare’s role from passive infrastructure to active negotiator in the AI data economy. As AI assistants become more independent, and as AI-generated answers increasingly replace traditional web traffic, tools like Pay per Crawl could determine which sites get fairly compensated.
(This article has been curated by Arfan Jeelany, who is an intern with The Indian Express)
© IE Online Media Services Pvt Ltd
0
0
Average Rating