ClaudeBot
ClaudeBot is Anthropic's training crawler. Here is what it does, its user-agent string, and how to allow or block it in robots.txt.
What is ClaudeBot?
ClaudeBot is Anthropic's web crawler. In Anthropic's words, it “helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.” It is Anthropic's training and data-collection crawler.
These bots collect web content to train future AI models. Blocking them keeps your content out of training data — it costs you no traffic, because training crawlers never send visitors.
The ClaudeBot user-agent string
This is the user-agent string Anthropic documents for ClaudeBot. You will see it in your server logs when the bot visits.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Anthropic's help page documents the robots.txt token “ClaudeBot” but does not publish the full Mozilla/5.0 string; the string above is the one widely observed in server logs.
How do I block ClaudeBot in robots.txt?
Add one of these snippets to the robots.txt file at the root of your domain. An explicit group for ClaudeBot overrides your User-agent: * rules for this bot.
Block ClaudeBot
Tells ClaudeBot it may not access any page on your site.
User-agent: ClaudeBot Disallow: /
Allow ClaudeBot
Explicitly allows ClaudeBot, even when a broad Disallow rule blocks other bots.
User-agent: ClaudeBot Allow: /
Does ClaudeBot respect robots.txt?
Yes, explicitly. Anthropic states: “Anthropic's Bots respect 'do not crawl' signals by honoring industry standard directives in robots.txt.” A User-agent: ClaudeBot group is honored.
Should you block ClaudeBot?
Blocking ClaudeBot is a fair choice if you do not want your content used to train Claude. It costs you no traffic. Anthropic exposes ClaudeBot, Claude-User, and Claude-SearchBot as separate robots.txt tokens, so you can block training while staying visible to Claude's search and assistant features.
Official documentation
The facts on this page come from Anthropic's crawler help page. Bot behavior changes — when in doubt, the operator's page is the source of truth:
https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawlerWhat does your robots.txt say about ClaudeBot?
Run your domain through our free checker to see whether ClaudeBot — and 13 other AI crawlers — may access your site right now.