The foremost web Content material Supply Community (CDN), Cloudflare, has declared warfare on AI corporations. Beginning July 1, Cloudflare now blocks by default AI net crawlers accessing content material out of your web sites with out permission or compensation.
The change addresses an actual downside. My very own small website, the place I observe all my tales, Sensible Expertise, has been slowed dramatically at occasions by AI crawlers. It is not simply me. Quite a few web site homeowners have reported that AI crawlers, reminiscent of OpenAI’s GPTBot and Anthropic’s ClaudeBot, generate large volumes of automated requests that clog up web sites so that they’re as sluggish as sludge. GoogleBot alone studies that the cloud-hosting service Vercel bombards the websites it hosts with over 4.5 billion requests a month.
These AI bots typically crawl websites much more aggressively than conventional search engine crawlers. They often revisit the identical pages each few hours and even hit websites with a whole bunch of requests per second. Whereas the AI corporations deny that their bots are responsible, the proof tells a special story.
Thus, on behalf of its two million-plus prospects, 20% of the online, Cloudflare now blocks AI crawlers. For any new web site signing up for its companies, AI crawlers can be mechanically blocked from accessing its content material except the location proprietor grants express permission. Moreover, Cloudflare guarantees to detect “shadow” scrapers — bots that try to evade detection — by utilizing behavioral evaluation and machine studying. What’s good for the AI goose is nice for the gander.
This transfer reverses the earlier established order, the place web site homeowners needed to decide out of AI crawling. Now, blocking is the default, and AI distributors should request entry and make clear their intentions, whether or not for mannequin coaching, search, or different makes use of, earlier than they’re allowed in.
This alteration arises not solely due to pissed off web site homeowners. Quite a few publishing corporations, reminiscent of The Related Press, Condé Nast, and ZDNET’s personal mum or dad firm, Ziff Davis, are pissed off that AI corporations have been “strip mining” the online for content material. All too typically, this has been completed with out compensation or consent, and generally, ignoring customary protocols like robots.txt that should block crawlers.
(Disclosure: Ziff Davis, ZDNET’s mum or dad firm, filed an April 2025 lawsuit towards OpenAI, alleging it infringed Ziff Davis copyrights in coaching and working its AI techniques.)
Furthermore, latest courtroom instances have dominated in favor of Meta and Anthropic, discovering that their use of copyrighted works was authorized underneath the doctrine of truthful use. For sure, writers, artists, and publishers don’t love this one bit. Publishers are nonetheless nervous that the federal authorities will give AI free rein to do because it needs with their content material. AI powerhouses reminiscent of OpenAI and Google are persevering with to foyer the federal government to categorise AI coaching on copyrighted information as truthful use.
It is also price noting that after the Copyright Workplace launched a pre-publication model of its 108-page copyright and AI report, which struck a center floor by supporting each of those world-class industries that contribute a lot to our financial and cultural development. Nonetheless, it added that whereas some generative AI in all probability constitutes a “transformative” use, the mass scraping of all information didn’t qualify as truthful use. The subsequent day, the Trump administration fired the head of the Copyright Workplace and changed her with an lawyer with no prior expertise in copyright legislation.
Given all this, it is no surprise that publishers sought an ally in know-how.
As Cloudflare CEO Matthew Prince mentioned in an announcement, its new coverage is supposed to “give publishers the management they deserve and construct a brand new financial mannequin that works for everybody—creators, shoppers, tomorrow’s AI founders, and the way forward for the online itself.”
To enhance the transfer to dam AI crawlers, Cloudflare has additionally launched its “Pay Per Crawl” program. This allows publishers to set their very own charges for AI corporations that need to scrape their content material.
This method is at the moment in non-public beta and goals to create a framework the place AI corporations will pay for entry, or be denied in the event that they refuse. Technically, this can be completed by dusting off an previous, principally unused net server response, HTTP 402, which responds with a “Fee Required” error message. This implies it needs to be easy to implement and appropriate with present web sites and their infrastructure.
General, it is a massive deal. Due to Cloudflare powering such a big portion of the web, a big quantity of net content material might develop into inaccessible to AI corporations except they negotiate entry or pay licensing charges. As Nicholas Thompson, CEO of The Atlantic, famous, “Till now, AI corporations haven’t wanted to pay for content material licenses as a result of they may merely take it with out repercussions. Now they might want to negotiate.”
Thus far, most AI corporations have been actively towards paying for content material. As Sir Nick Clegg, former deputy UK Prime Minister and Meta govt, mentioned not too long ago, merely asking artists’ permission earlier than they scrape copyrighted content material will “principally kill the AI trade.”
Cloudflare’s new coverage is a direct response to this strategy and the rising quantity and intrusiveness of AI crawlers which have include it. It is also an try to cease the siphoning of visitors that may in any other case go to publishers.
Because the rise of AI, visitors to information websites has plunged. For instance, Enterprise Insider’s visitors dropped by over half, 55% from April 2022 to April 2025. Left unchecked, Thompson not too long ago predicted that, because of AI, the Atlantic workers ought to count on visitors from Google to drop to zero.
What’s going to occur subsequent? Will the opposite CDN, reminiscent of Akamai, comply with swimsuit? Keep tuned. For now, the period of unrestricted AI crawling seems to be ending, effectively, no less than for the fifth of the web that flows by means of Cloudflare’s pipes.
Get the morning’s prime tales in your inbox every day with our Tech At present publication.
