What Is ClaudeBot?
Updated 21 September 2026
Jump to section
ClaudeBot is Anthropic's web crawler, and it collects content to help train the Claude AI models. It is one of three Anthropic bots, each with a separate job, and all three respect the robots.txt rules a site sets. Blocking ClaudeBot limits what future Claude models learn from your site. On its own, it does not remove you from the answers Claude gives when a user asks it to open a live page.
What does ClaudeBot do?
ClaudeBot gathers web content for model training, and nothing else.
Anthropic's April 2026 documentation says ClaudeBot "helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training."
Training is a slow loop. What ClaudeBot collects today may shape a future model's knowledge, but it does not change the answer Claude gives this week.
That gap is why the bot's name causes confusion. Most questions about ClaudeBot are really questions about live citations, which belong to a different crawler and the live retrieval behind it.
The three Anthropic crawlers
Anthropic runs three bots, each doing one job. The names below are the user-agent tokens that appear in robots.txt rules and server logs.
| Bot | Its job |
|---|---|
| ClaudeBot | Collects web content to train Claude models |
| Claude-User | Fetches a page when a user asks Claude about it |
| Claude-SearchBot | Navigates the web to improve Claude's search results |
The search bot is the one tied to live citations. Anthropic says Claude-SearchBot "navigates the web to improve search result quality for users."
So a page it can read is a page Claude can surface when it searches. That is separate from training, and controlled separately.
Does ClaudeBot respect robots.txt?
Yes. Anthropic says its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt."
It adds that they support the non-standard Crawl-delay directive too, which lets a site slow how often they visit.
That is a contrast with one OpenAI bot. GPTBot's stablemate ChatGPT-User is documented as possibly ignoring robots.txt, because a person triggers each fetch. Anthropic makes no such carve-out for its own user-triggered bot.
Should you block ClaudeBot?
Blocking ClaudeBot is a real decision, not a default.
For most Malaysian SMEs that want to be found, leaving it open is the better call. Training familiarity is part of how a brand becomes something an assistant simply knows. Blocking makes sense mainly for publishers protecting content they sell.
The first thing we check on a site that wants AI mentions is its own robots.txt. A single inherited block can quietly shut out ClaudeBot, GPTBot, Google-Extended, and PerplexityBot at once.
Whether to open or close each door is the wider question in should you block AI crawlers.
Frequently asked questions
Is ClaudeBot the same as ChatGPT's crawler?
No. ClaudeBot belongs to Anthropic and feeds the Claude models. ChatGPT's crawlers belong to OpenAI, and GPTBot is the OpenAI training bot. The two companies run separate crawlers with separate robots.txt tokens, so a rule for one does nothing to the other.
Does ClaudeBot respect robots.txt?
Yes, and it goes a step further than some bots. Anthropic documents that its crawlers honor robots.txt "do not crawl" directives, and that they also support Crawl-delay. So you can block ClaudeBot entirely, or simply ask it to visit less often. A block is a request the standard expects it to honor, not a technical wall.
Should I block ClaudeBot?
Usually not, if you want AI visibility. Training data is part of how a brand becomes something Claude knows without searching. Blocking it works against a site whose purpose is to be found. The clearest case for blocking is a publisher protecting paid content.
What is the difference between ClaudeBot and Claude-SearchBot?
ClaudeBot collects content for training future Claude models. Claude-SearchBot reads the web so Claude can surface current pages when it searches. One shapes what the model knows over time; the other affects what it can cite live. They are controlled separately in robots.txt.
How do I know if ClaudeBot has visited my site?
Check your server logs or hosting statistics for the ClaudeBot user agent. Most hosting control panels can filter visits by user agent, and the token appears plainly in each request. Seeing it confirms collection for training. It says nothing about whether Claude cites you, which you test by asking Claude your customers' questions.
Deciding which AI crawlers to welcome
Storming Solutions runs SEO, AEO, and GEO for Malaysian businesses from Kuala Lumpur, and crawler access is part of that technical groundwork. We think most blanket AI blocks were set by a plugin default, not by the business, and never revisited.
Not sure what your own robots.txt lets ClaudeBot and the others do? Ask us through our SEO, AEO and GEO service, or message us on WhatsApp. The free AI Visibility Report includes a check of which AI crawlers your site currently admits, alongside your core SEO in Malaysia setup.