Claude-User
Claude-User fetches a page when a Claude user asks about it. Here is what it does, its user-agent string, and how robots.txt controls it.
What is Claude-User?
Claude-User is Anthropic's user-triggered fetcher. When someone asks Claude a question, it may visit websites on that user's behalf using the Claude-User agent to help answer. It is not a bulk crawler and is not used for training.
These bots fetch a page live when someone asks an AI assistant about it. Blocking them means assistants like ChatGPT and Claude cannot read or cite your pages on demand.
The Claude-User user-agent string
This is the user-agent string Anthropic documents for Claude-User. You will see it in your server logs when the bot visits.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)
Anthropic's help page documents the robots.txt token “Claude-User” but does not publish the full Mozilla/5.0 string; the string above is the one widely observed in server logs.
How do I block Claude-User in robots.txt?
Add one of these snippets to the robots.txt file at the root of your domain. An explicit group for Claude-User overrides your User-agent: * rules for this bot.
Block Claude-User
Tells Claude-User it may not access any page on your site.
User-agent: Claude-User Disallow: /
Allow Claude-User
Explicitly allows Claude-User, even when a broad Disallow rule blocks other bots.
User-agent: Claude-User Allow: /
Does Claude-User respect robots.txt?
Yes. Anthropic's statement that its bots “respect 'do not crawl' signals by honoring industry standard directives in robots.txt” covers Claude-User too. This is notably stricter than OpenAI's ChatGPT-User, for which robots.txt “may not apply.”
Should you block Claude-User?
There is usually little reason to block Claude-User: it only fetches a page when one of your visitors, through Claude, explicitly asks about it. Because Anthropic honors robots.txt for this agent, you can block it cleanly if you prefer — but doing so stops Claude from reading your public pages on a user's request.
Official documentation
The facts on this page come from Anthropic's crawler help page. Bot behavior changes — when in doubt, the operator's page is the source of truth:
https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawlerWhat does your robots.txt say about Claude-User?
Run your domain through our free checker to see whether Claude-User — and 13 other AI crawlers — may access your site right now.