Google-Extended
Google-Extended is a robots.txt token, not a crawler. Here is what it controls, why it has no user agent, and how to opt out of Gemini AI training.
What is Google-Extended?
Google-Extended is not a crawler — it is a robots.txt control token. It lets you decide whether content Google has already crawled may be used to train Gemini models and ground answers in Gemini Apps and the Vertex AI API. Google states it “does not impact a site's inclusion in Google Search nor is it used as a ranking signal.”
These bots collect web content to train future AI models. Blocking them keeps your content out of training data — it costs you no traffic, because training crawlers never send visitors.
The Google-Extended user-agent string
Google-Extended has no separate user-agent string. Google's docs state crawling is done with existing Google user agents; the Google-Extended token is used only as a robots.txt control, so you will never see “Google-Extended” in your server logs as a fetching agent.
How do I block Google-Extended in robots.txt?
Add one of these snippets to the robots.txt file at the root of your domain. An explicit group for Google-Extended overrides your User-agent: * rules for this bot.
Block Google-Extended
Tells Google-Extended it may not access any page on your site.
User-agent: Google-Extended Disallow: /
Allow Google-Extended
Explicitly allows Google-Extended, even when a broad Disallow rule blocks other bots.
User-agent: Google-Extended Allow: /
Does Google-Extended respect robots.txt?
It exists purely as a robots.txt control token. Adding a User-agent: Google-Extended / Disallow: / group opts your already-crawled content out of Gemini training and grounding — without affecting Google Search indexing or ranking.
Should you block Google-Extended?
Disallowing Google-Extended is a low-risk way to opt out of Gemini AI training while keeping your normal Google Search visibility fully intact — Google is explicit that the token has no effect on Search. If you want to stay in Search but out of AI training, this is exactly the token to set.
Official documentation
The facts on this page come from Google's crawler documentation. Bot behavior changes — when in doubt, the operator's page is the source of truth:
https://developers.google.com/search/docs/crawling-indexing/google-common-crawlersWhat does your robots.txt say about Google-Extended?
Run your domain through our free checker to see whether Google-Extended — and 13 other AI crawlers — may access your site right now.