What Is Cloudflare's AI-Crawler Blocking (Pay Per Crawl)?

Cloudflare's AI-crawler controls let a website owner allow, block, or charge the bots that AI companies use to read the web. In July 2025 Cloudflare said it was changing the default to block AI crawlers unless they pay. It also introduced "pay per crawl," which uses the HTTP 402 code to bill a crawler per request. Since 15 September 2026, its Block setting also stops Googlebot, so refuse AI training with Disallow AI Training instead.

What Cloudflare's AI-crawler blocking is

Cloudflare's AI-crawler blocking is a control that stops AI bots from reading a site unless the owner allows them. Cloudflare is a large infrastructure provider that sits in front of many websites, so this control reaches a lot of the web.

In a July 2025 announcement, Cloudflare said it was "changing the default to block AI crawlers unless they pay creators for their content". The stated aim was to give creators a say over whether AI companies use their work.

The important word is default. A preset you never picked can still decide which AI tools read your site, so it pays to know which one applies.

Google needs care. Since Cloudflare's 15 September 2026 update, its Block setting also stops Googlebot, while Disallow AI Training keeps search and refuses training. That setting writes robots.txt rules, the same route as tokens like Google-Extended.

What pay per crawl is

Pay per crawl is a Cloudflare system that lets a site charge AI crawlers a fee for each request. Cloudflare introduced it in private beta in July 2025, describing it as an early experiment in paid access. In September 2026 Cloudflare added Pay Per Use, in beta, which pays publishers when their content is actually used.

It works through standard web plumbing. Cloudflare says pay per crawl uses "HTTP status codes and established authentication mechanisms to create a framework for paid content access," built on the HTTP 402 "Payment Required" response.

For each AI crawler, an owner picks one of three options.

Setting	What it does
Allow	Give the crawler free access to the content
Charge	Require payment at the price you set
Block	Deny access entirely, with no option to pay

So the choice is not simply on or off. An owner can let some bots in free, charge others, and block the rest, all from one place.

Why this matters for your AI visibility

Cloudflare's controls matter because a block can remove you from the raw material AI learns from and answers with. If AI crawlers cannot read your pages, they cannot learn about you before a model's knowledge cutoff, and some cannot cite you at answer time.

That is the same openness argument behind Common Crawl and the individual crawlers like GPTBot. A page that no AI bot can reach is a page no AI model can learn or quote.

There is a real trade-off here, which is the whole point of the tool. Blocking protects your content from being used without payment. It also risks making you invisible to the AI answers customers increasingly read first.

Which way to lean is a genuine decision, not a default to accept blindly. We walk through it in whether you should block AI crawlers, where the short version for most businesses is that visibility beats blocking.

What to check on your own site

Start by finding out whether your site even runs on Cloudflare, because many owners do not know. If your site's traffic is proxied through Cloudflare, its bot settings apply to you, and a preset default may already be in force.

Then check the AI-crawler setting, not just the general firewall. Choose Disallow AI Training to refuse training but keep Google, because Block now stops Googlebot too.

Decide on purpose. Choose allow, disallow training, charge, or block because it fits your goals, not because a default chose for you.

In our experience, the first thing to check on a site that wants AI mentions is whether something is blocking the crawlers it needs. On a Cloudflare site, that check covers the AI-bot setting, not just robots.txt.

Frequently asked questions

Does Cloudflare block Google or hurt my normal SEO?

It can, depending on the setting. Since 15 September 2026, Cloudflare's Block and "Block on pages with ads" settings also stop Googlebot, Bingbot, and Applebot, which affects normal search. To refuse AI training but stay in Google, choose Disallow AI Training. Cloudflare says Apple, Google, and Microsoft honor it or have committed to.

Is AI-crawler blocking on by default for my site?

It depends on when your site joined and whether it runs ads. Since 15 September 2026, Cloudflare offers new domains presets: search stays allowed, and ad-funded sites get Disallow AI Training plus agent blocking on ad pages. Existing domains had older settings migrated. The only reliable way to know is to log in and check your bot settings directly. Do not assume; a default block is exactly the kind of thing that hides a site from AI without warning.

Can I really charge AI companies to crawl my site?

In principle, yes. Cloudflare's pay per crawl lets an owner set a price and bill AI crawlers per request using the HTTP 402 code. It launched in private beta in July 2025, and in September 2026 Cloudflare added Pay Per Use, a beta that pays when content is used rather than crawled.

Both suit large publishers more than a small business. For most SME sites, the practical decision is still allow, disallow training, or block.

If I block AI crawlers, will ChatGPT stop mentioning me?

Over time, it can. If AI crawlers cannot read your site, they have less to learn from and, for tools that cite live pages, less to quote. You may still be mentioned from other sources that talk about you, but you lose control of your own best material. That is why blocking is a real trade-off, tied to where these tools get their answers in the first place.

Do I need Cloudflare to manage AI crawlers?

No. Cloudflare is one convenient way to control AI bots, but you can also manage them through your robots file and server rules without it. What matters is that some control point exists and that you have set it deliberately. If you already use Cloudflare as your CDN, its AI-bot settings are simply the easiest place to make that choice.

Choose your setting on purpose

Cloudflare's AI-crawler controls hand website owners a real lever over who reads their content, from a free allow to a hard block to a per-request charge. The risk is not the tool; it is leaving a default in place that you never chose. Find out whether your site is affected, decide what fits your goals, and set it on purpose.

Storming Solutions runs SEO, AEO, and GEO for Malaysian businesses from Kuala Lumpur. Checking whether AI systems can actually reach a site is one of the first things we look at. We would rather you choose to be found than discover a default chose otherwise.

Not sure whether AI crawlers can read your site? Our SEO, AEO and GEO service audits exactly that, and you can message us on WhatsApp to start.