By Andrew Griffin
Cloudflare, the internet infrastructure company that powers much of the web, says that it is “changing the rules of the internet”.
The company will block AI crawlers by default in an attempt to stop artificial intelligence companies from gathering data from across the web.
Large language models, such as ChatGPT, require vast amounts of training data in order to improve their performance. Many of those companies have responded by using crawlers to access different web pages and store their contents so that it can be used to train those systems.
Now, Cloudflare says that those AI crawlers will be blocked by default. That means those automated systems should not be able to visit pages without “permission or compensation”, it said.
The publishers of those websites will be able to give AI crawlers permission to access their sites. And Cloudflare says that it will add the option for a “pay by crawl” fee, which AI companies will be able to choose whether to pay.
The use of online writing to train AI systems has become one of the most divisive issues of the ongoing era of artificial intelligence. A number of publishers have become locked in legal action against AI companies that they argue have wrongly taken their writing to train their systems.
As that conflict increased, in 2023, Cloudflare initially said that it would allow websites to add a special tag to their sites to ask AI websites not to access them, but it was unenforceable. Last year, it started to allow websites to automatically block them, using technology that spotted such AI bots.
Now, that latter tool will be on by default.
Matthew Prince, Cloudflare’s chief executive, told the New York Times that it was “changing the rules of the internet across all of Cloudflare”. The move was made from a concern that AI companies scraping the web and accessing content freely would stop people being incentivised from publishing new writing or other forms of content, he said.