Cloudflare will now block AI crawlers on your website – and even force them to pay you – 1v1 Video Chat & LIve Streaming & Influencer Subscription

By Benedict Collins

Tech Radar Pro

Tech Radar Gaming

Tech Radar Pro

TechRadar the business technology experts

Search TechRadar

View Profile

België (Nederlands)

Deutschland

North America

US (English)

Australasia

New Zealand

Expert Insights

Website builders

Web hosting

Best website builder
Best web hosting
Best office chairs
Best antivirus
Expert Insights

You may like

GenAI bots could well be scraping your web apps, researchers warn

Forget Chrome and Edge – this challenger browser now offers greater protection from online scams

I tested PageCrawl, a great free website content monitoring tool

Charging AI crawlers for access
HTTP response code 402, otherwise known as the 402 Payment Required status code, indicates to crawlers payment is needed to access the content. As a result, the crawler can either respond with intent to pay, or is blocked from accessing the content.

As an added bonus, content creators with a block on their site can effectively ‘tell’ AI crawlers that they are open to potential payments in the future.

(Image credit: Cloudflare)
For those thinking that someone could simply spoof a crawler that has access to the site, Cloudflare is one step ahead. An authentic crawler will use the ‘signature-agent’, ‘signature-input’, and ‘signature’ headers to authenticate themselves with Cloudflare.
Cloudflare will then compare a public key from a Ed25519 key pair that is stored in a hosted directory with the URL of the key directory and user agent information that is registered with Cloudflare, thus allowing the authentic crawler through and blocking any spoofed crawlers.

Are you a pro? Subscribe to our newsletter
Sign up to the TechRadar Pro newsletter to get all the top news, opinion, features and guidance your business needs to succeed!
Contact me with news and offers from other Future brandsReceive email from us on behalf of our trusted partners or sponsorsBy submitting your information you agree to the Terms & Conditions and Privacy Policy and are aged 16 or over.

(Image credit: Cloudflare)
Crawlers will also be able to crawl the web with a set budget for accessing protected sites using the ‘crawler-exact-price’ header to accept the proposed price listed by the ‘crawler-price’ header on the desired site, or preemptively use the ‘crawler-max-price’ when accessing a site which will grant access if the price is equal to or less than the crawler’s budget.
Cloudflare also has some theories for the potential of pay per crawl in the future. An AI agent can be given a budget to crawl the web when responding to a prompt, allowing the user to access high-quality and relevant content when entering a prompt.
Pay per crawl is currently only available in private beta, but interested parties can reach out to Cloudflare via the link at the bottom of the blog.
You might also like

These are the best web scraping APIs
This mega DDoS attack broke all records
We’ve reviewed all the best cloud storage

Benedict Collins

Social Links Navigation
Senior Writer, Security

Benedict has been writing about security issues for over 7 years, first focusing on geopolitics and international relations while at the University of Buckingham. During this time he studied BA Politics with Journalism, for which he received a second-class honours (upper division),then continuing his studies at a postgraduate level, achieving a distinction in MA Security, Intelligence and Diplomacy. Upon joining TechRadar Pro as a Staff Writer, Benedict transitioned his focus towards cybersecurity, exploring state-sponsored threat actors, malware, social engineering, and national security. Benedict is also an expert on B2B security products, including firewalls, antivirus, endpoint security, and password management.

You must confirm your public display name before commenting

Please logout and then login again, you will then be prompted to enter your display name.

GenAI bots could well be scraping your web apps, researchers warn

Forget Chrome and Edge – this challenger browser now offers greater protection from online scams

I tested PageCrawl, a great free website content monitoring tool

Chrome has a new AI-powered ‘enhanced protection’ mode to protect you from scams – here’s how it works

A huge online fraud operation is hijacking WordPress sites to send out 1.4 billion ad requests per day

Fake Cloudflare CAPTCHA page laden with malware uncovered in the wild – here’s how to stay secure and safe

Latest in Pro

Six reasons why today’s AI infrastructure is unsustainable

This could be a great time to buy a new work PC – no-one is buying the devices makers stockpiled in anticipation of Trump tariffs

Qantas reveals massive data breach – 6 million customers possibly affected, here’s what we know

Google’s data centers using more power than ever before as AI surge continues

The Human Firewall: even with AI, humans are still the last line of defense in cybersecurity

WhatsApp will now let big businesses voice call you directly, also adds more AI agents

Latest in News

Microsoft just gave the Windows 11 Notepad app a controversial feature that people are either loving or hating

iOS 26 will soon give your AirPods a smart sleep trick – and Beats headphones could join the pajama party

Dying Light: The Beast won’t have branching narrative choices, but the developer says it will pave the way for the series’ future

This could be a great time to buy a new work PC – no-one is buying the devices makers stockpiled in anticipation of Trump tariffs

Everything new on Max in July 2025 – stream 78 new movies, including Sinners, Death of a Unicorn and more

iPhone 17 Air leaked specs suggest it’s likely to beat the S25 Edge in one key way – and lose out in another