By Jess Kinghorn
Skip to main content
PC Gamer THE GLOBAL AUTHORITY ON PC GAMES
Search PC Gamer
View Profile
Movies & TV
Gaming Industry
PC Gaming Show
Newsletter Signup
Community Guidelines
Affiliate Links
Meet the team
About PC Gamer
PC Gamer Magazine Subscription
Why subscribe?
Subscribe to the world’s #1 PC gaming mag
Try a single issue or save on a subscription
Issues delivered straight to your door or device
From£35.99View
Essential Hardware
PC Gaming Show
Dune: Awakening
Recommended reading
‘Widespread theft’: The UK government’s fifth attempt to push through a bill allowing AI companies to scrape any data they like shut down by the House of Lords
An AI holds the top slot in a leaderboard that ranks people who hunt for system vulnerabilities used by hackers
US judge rules that Anthropic’s use of copyrighted content to train AI was fair use, but pirating books is step too far
Anthropic says its Claude AI will resort to blackmail in ‘84% of rollouts’ while an independent AI safety researcher also notes it ‘engages in strategic deception more than any other frontier model that we have previously studied’
Microsoft pulls out of two big data centre deals because it reportedly doesn’t want to support more OpenAI training workloads
Google search’s AI overviews are awful, but here’s a browser extension that gets rid of them
The LA Times gives mealy-mouthed AI the last word on its opinion pieces
Cloudflare will block AI web crawlers by default, and introduces new Pay Per Crawl scheme that means AI companies will have to fork out for the privilege to scrape
Jess Kinghorn
2 July 2025
The bill has been long due.
When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works.
(Image credit: David Paul Morris/Bloomberg via Getty Images)
As time goes on, just saying ‘no’ to AI feels more and more futile. However, Cloudflare is announcing a few more tools for your anti-AI arsenal that put some of the power back in your hands. Cloudflare offers a range of network services from domain registration and content delivery to DDOS mitigation. Adding to that quiver, Cloudflare is launching the sharp and pointy Pay Per Crawl scheme, which aims to hit AI companies scraping online content where it hurts—namely, their deep pockets.
Last year, Cloudflare began allowing customers to simply block AI bots from their websites with one click. This AI Audit tool also allows users to pick and choose which bots they want to block, as well as offering insights into which LLM or gen-AI is regurgitating their content. Now, Cloudflare has announced that AI crawlers will be blocked by default, describing it as “the first step toward a more sustainable future for both content creators and AI innovators.”
Users who want to have their content scraped for free can still turn this default setting off, but Cloudflare’s second announcement seemingly asks the question, ‘Why would you want to?’
Related Articles
‘Widespread theft’: The UK government’s fifth attempt to push through a bill allowing AI companies to scrape any data they like shut down by the House of Lords
An AI holds the top slot in a leaderboard that ranks people who hunt for system vulnerabilities used by hackers
US judge rules that Anthropic’s use of copyrighted content to train AI was fair use, but pirating books is step too far
Pay Per Crawl, a scheme announced last year that allows users to charge AI companies wanting to scrape their content, is now in beta. To date, Cloudflare has not revealed the names of any Pay Per Crawl participants aside from ProRata, the startup behind AI search engine Gist.AI.
Founder and CEO Bill Gross offered comment as part of Cloudflare’s announcement regarding AI crawlers, saying, “Our whole mission is to protect and elevate human creativity in the AI era. That’s why we’re proud to be one of the first AI companies to participate in Cloudflare’s initiative to create a new permission-based model for the Internet. We believe that creators and publishers deserve to be fairly compensated for the value they bring, and we’re thrilled to support Cloudflare in helping make that vision a reality.”
It doesn’t seem unlikely that bigger AI companies would go for a scheme like this. After all, OpenAI has already struck content partnerships with a number of media companies, including PC Gamer’s own parent company, Future. Pay Per Crawl feels like an attempt to level the playing field, giving smaller content creators a leg to stand on, who would otherwise simply not have an entire team to dedicate to working out the terms of a content partnership.
But let’s reverse for a second, especially because, as Wired notes, not all web crawlers are bad. For instance, the Internet Archive’s preservation work in particular relies heavily on them. Last year, the Pew Research Center found that a quarter of all webpages that existed between 2013 and 2023 are now no longer accessible. Web crawlers are one way to preserve pages like that before they disappear in a puff of broken links and long-forgotten hosting services.
The biggest gaming news, reviews and hardware deals
Keep up to date with the most important stories and the best deals, as picked by the PC Gamer team.
Contact me with news and offers from other Future brandsReceive email from us on behalf of our trusted partners or sponsorsBy submitting your information you agree to the Terms & Conditions and Privacy Policy and are aged 16 or over.
AI crawlers, however, are a bit of a menace as they scrape without permission from the owner of the content, and can end up overly taxing servers by behaving not unlike a DDOS attack. As previously mentioned, Cloudflare is no stranger to DDOS attacks, telling our Nick last month that their network blocks about ‘225,000,000,000 attacks per day.’ The company also shared that its network recently mitigated the largest attack ever recorded, battering a single IP address with 37.5 TB of data over the course of 45 seconds.
As such, Cloudflare’s AI blocking tools leverage a mix of machine learning and behavioural analysis to differentiate between these two types of web crawlers. Cloudflare has also found some success fingerprinting naughty bots by punting AI web crawlers down a rabbit warren of AI-generated pages. Now that’s what I call fighting fire with fire…or a flinging slop fight. At any rate, would it be naive to feel just a little optimistic about Cloudflare’s push for a ‘Permission-Based Model for the Internet’? Time will tell.
Best gaming PC 2025All our current recommendations
👉Check out our full guide👈
1. Best overall: HP Omen 35L
2. Best budget: Lenovo Legion Tower 5i
3. Best compact: Velocity Micro Raptor ES40
4. Alienware: Alienware Aurora
5. Best mini PC: Minisforum AtomMan G7 PT
Jess Kinghorn
Social Links Navigation
Hardware Writer
Jess has been writing about games for over ten years, spending the last seven working on print publications PLAY and Official PlayStation Magazine. When she’s not writing about all things hardware here, she’s getting cosy with a horror classic, ranting about a cult hit to a captive audience, or tinkering with some tabletop nonsense.
You must confirm your public display name before commenting
Please logout and then login again, you will then be prompted to enter your display name.
‘Widespread theft’: The UK government’s fifth attempt to push through a bill allowing AI companies to scrape any data they like shut down by the House of Lords
An AI holds the top slot in a leaderboard that ranks people who hunt for system vulnerabilities used by hackers
US judge rules that Anthropic’s use of copyrighted content to train AI was fair use, but pirating books is step too far
Anthropic says its Claude AI will resort to blackmail in ‘84% of rollouts’ while an independent AI safety researcher also notes it ‘engages in strategic deception more than any other frontier model that we have previously studied’
Microsoft pulls out of two big data centre deals because it reportedly doesn’t want to support more OpenAI training workloads
Google search’s AI overviews are awful, but here’s a browser extension that gets rid of them
Latest in AI
A ‘big beautiful bill’ provision to prevent states from regulating AI for 10 years got nixed by the US Senate
Anthropic tasked an AI with running a vending machine in its offices, and it not only sold some products at a big loss but it invented people, meetings, and experienced a bizarre identity crisis
Accused of being AI after racking up well over 400,000 monthly Spotify listens, The Velvet Sundown claims they are ‘a real band’ and ‘never use AI’
MrBeast scraps AI YouTube thumbnail generator days after announcing it: ‘If creators don’t want the tools, no worries’
Ring is using AI to generate video descriptions of what goes on outside your door—and to keep even more detailed tabs on ‘the routines of your residence’
‘Artificial intelligence is going to displace millions and millions of workers’ says Bernie Sanders, so might as well take a four-day week
Latest in News
Free-to-play mech brawler Mecha Break has attracted 50,000 players within 10 minutes of launching, proving our appetite for giant robots remains insatiable
A new WoW event boosts ultra-rare item drop rates tenfold, which admittedly isn’t a huge difference when the starting number is 0.5%
Hideo Kojima goes in hard on AI and is worried we’ll all ‘be unknowingly led into a predetermined lifestyle’
Destiny 2 just got weird: Launch trailer leans hard into time travel and looks more like a Control crossover
‘It makes me cringe every time’: 25 years after Diablo 2 reshaped the RPG landscape, Diablo creator David Brevik still thinks its stamina bar sucks
A ‘big beautiful bill’ provision to prevent states from regulating AI for 10 years got nixed by the US Senate
HARDWARE BUYING GUIDES
LATEST GAME REVIEWS
Best Steam Deck accessories in Australia for 2025: Our favorite docks, powerbanks and gamepads
Best graphics card for laptops in 2025: the mobile GPUs I’d want in my next gaming laptop
Best mini PCs in 2025: The compact computers I love the most
Best 14-inch gaming laptop in 2025: The top compact gaming laptops I’ve held in these hands
Best Mini-ITX motherboards in 2025: My pick from all the mini mobo marvels I’ve tested
Sennheiser Momentum True Wireless 4 review
Razer Joro & Basilisk Mobile review
Glorious Model O Eternal review
LaCie Rugged Pro 5 SSD review
Seagate Ultra Compact review
PC Gamer is part of Future plc, an international media group and leading digital publisher. Visit our corporate site.
Contact Future’s experts
Terms and conditions
Privacy policy
Cookies policy
Advertise with us
Accessibility Statement
Future Publishing Limited Quay House, The Ambury,
BA1 1UA. All rights reserved. England and Wales company registration number 2008885.
Please login or signup to comment
Please wait…