Claude-SearchBot
Claude-SearchBot indexes the web to improve Claude's search answers. Here is what it does, its user-agent string, and how to allow or block it.
What is Claude-SearchBot?
Claude-SearchBot is Anthropic's search crawler. It “navigates the web to improve search result quality for users,” analyzing online content to make Claude's search responses more relevant and accurate. It is used for search quality, not model training.
These bots index your site for AI-powered search engines that cite and link back to you. Blocking them makes you invisible exactly where a growing number of people now ask their questions.
The Claude-SearchBot user-agent string
This is the user-agent string Anthropic documents for Claude-SearchBot. You will see it in your server logs when the bot visits.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +Claude-SearchBot@anthropic.com)
Anthropic's help page documents the robots.txt token “Claude-SearchBot” but does not publish the full Mozilla/5.0 string; the string above is the one widely observed in server logs.
How do I block Claude-SearchBot in robots.txt?
Add one of these snippets to the robots.txt file at the root of your domain. An explicit group for Claude-SearchBot overrides your User-agent: * rules for this bot.
Block Claude-SearchBot
Tells Claude-SearchBot it may not access any page on your site.
User-agent: Claude-SearchBot Disallow: /
Allow Claude-SearchBot
Explicitly allows Claude-SearchBot, even when a broad Disallow rule blocks other bots.
User-agent: Claude-SearchBot Allow: /
Does Claude-SearchBot respect robots.txt?
Yes. It is covered by Anthropic's statement that its bots honor robots.txt directives, and it can be controlled independently through a User-agent: Claude-SearchBot group.
Should you block Claude-SearchBot?
Think twice before blocking Claude-SearchBot. It does not train models — it is how Claude finds and cites your site in search answers. Blocking it removes you from that surface without any of the training-data privacy benefit that blocking ClaudeBot already gives you.
Official documentation
The facts on this page come from Anthropic's crawler help page. Bot behavior changes — when in doubt, the operator's page is the source of truth:
https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawlerWhat does your robots.txt say about Claude-SearchBot?
Run your domain through our free checker to see whether Claude-SearchBot — and 13 other AI crawlers — may access your site right now.