Should You Block AI Crawlers?

No, most businesses should not block AI crawlers. Blocking them removes you from the answers customers now read first, on ChatGPT, Perplexity, and Google's AI results. There are narrow reasons to block a specific bot: a paywall, genuinely private data, or one crawler straining your server. But for a normal business site trying to be found, a blanket block quietly deletes you from the places buyers are starting to ask.

Should a normal business website block AI crawlers?

No. A normal business wants to be named and cited, and blocking the crawlers is the surest way not to be. The visibility you would give up is the whole point of the exercise.

Most sites that block are publishers protecting paid articles, not small businesses. Even the most-blocked bot is refused by a small minority of sites: GPTBot is blocked by about 5.89% of roughly 140 million websites (Ahrefs, data through December 2024). The other 94% stay readable on purpose.

The reason the default is "allow" is mechanical. An engine can only quote a page it is permitted to read, so a block is a decision to be absent from AI answers entirely.

It also forfeits any say in how those engines choose whom to cite.

Which crawlers would you even be blocking?

You would be blocking a handful of named bots, each with a different job, and lumping them together is the common mistake. Blocking the wrong one costs you visibility for no benefit.

OpenAI runs GPTBot for training and OAI-SearchBot for ChatGPT's live search. Anthropic runs ClaudeBot for training and Claude-SearchBot for search indexing. Google uses Googlebot for search, and a separate Google-Extended for Gemini's training and grounding (Google Search Central, checked July 2026).

That last one matters. Google states Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

Blocking it does not remove you from AI Overviews, which run on normal Search indexing instead. Those still follow your standard search snippet controls.

Crawler	Run by	Blocking it costs you
OAI-SearchBot	OpenAI	Citations in ChatGPT's live search
ClaudeBot / Claude-SearchBot	Anthropic	Presence in Claude's answers
Googlebot	Google	Search results and AI Overviews together
GPTBot / Google-Extended	OpenAI / Google	A future training snapshot only

When does blocking an AI crawler actually make sense?

Blocking makes sense in a few narrow cases, and selective blocking beats a blanket rule every time. The test is whether a specific bot causes a specific problem.

Three cases qualify. Paywalled or subscription content you sell rather than give away. Genuinely private sections, which should sit behind a login regardless. A single crawler hitting your server hard enough to slow the site for real visitors.

Even then, block only the offending bot, and keep the search crawlers allowed. Refusing an AI answer engine to "protect" a public marketing page trades your own visibility for nothing.

How do you control AI crawlers without disappearing?

You control them with robots.txt, the file that tells each named bot what it may read. The standard behind it, RFC 9309, lets you allow the search crawlers while refusing only a training bot if you object to that use.

The judgment call is training versus search. Allowing OAI-SearchBot keeps you citable in ChatGPT today; blocking GPTBot only withholds you from a future model. Those are separate decisions, and where ChatGPT gets its answers explains why the search path is the one that earns mentions now.

The first thing we check on a site chasing AI mentions is whether its own robots.txt already blocks the crawlers it wants. It is often a leftover from a template or a nervous developer, and fixing that one file is frequently the whole job.

From there, the work is ordinary getting mentioned by ChatGPT: pages an engine can read and wants to quote. A separate llms.txt file can offer guidance to the models that read it, without touching crawl access.

Frequently asked questions

Will blocking AI crawlers protect my content from being copied?

Not reliably. Blocking the well-behaved crawlers that respect robots.txt only hides you from the engines that would have cited you, while doing little against bad actors who ignore the rules. You lose the visibility and keep most of the risk. If specific content is truly sensitive, put it behind a login rather than relying on a crawler request you cannot enforce.

Does robots.txt actually stop the bots?

It stops the ones that choose to obey it, which includes the major AI crawlers from OpenAI, Anthropic, and Google. Anthropic's three crawlers respect robots.txt and crawl-delay, and Google's and OpenAI's search bots read it too. It is a polite instruction, not a wall, so a crawler built to ignore it can. For the mainstream engines that matter to your visibility, it is the correct control.

If I block GPTBot, do I still show up in ChatGPT?

Yes, usually. GPTBot only gathers pages to train future models, so blocking it does not remove you from ChatGPT's live search, which uses OAI-SearchBot instead. Many site owners block GPTBot to opt out of training while staying fully citable in live answers. To disappear from ChatGPT's answers entirely, you would have to block the search crawler too, which is rarely the goal.

Does blocking Google-Extended remove me from AI Overviews?

No. Google states plainly that Google-Extended governs Gemini's training and grounding, and does not affect your inclusion in Google Search or act as a ranking signal. AI Overviews are part of Search and run on normal indexing, so your standard snippet controls apply there. Blocking Google-Extended is an opt-out of Gemini training, nothing more.

Should I use llms.txt instead of blocking?

They do different jobs, so it is not an either-or. Blocking through robots.txt controls crawl access, while an llms.txt file offers content guidance to models that choose to read it. Neither is a formal standard every engine honors yet. For a business that wants to be found, the useful move is allowing the search crawlers first, then adding llms.txt as an optional extra.

The default is to be readable

Storming Solutions runs SEO, AEO, and GEO for Malaysian businesses from Kuala Lumpur, so "should we block these bots?" is a question we answer most weeks. Our honest view: for almost every business we work with, the goal is to be read by AI engines, not hidden from them. The rare exceptions are handled one bot at a time.

Not sure what your own robots.txt is telling the crawlers? That is one of the first things our free AI Visibility Report checks. Send us your site on WhatsApp or see how we approach it on our SEO, AEO & GEO service page.