Google-Extended

Google-Extended is best understood as a downstream-use switch in robots.txt, not as an AI crawler that visits a site under its own name.

1. What Google-Extended is

Google-Extended is a standalone product token accepted in a robots.txt User-agent: field. Google uses the resulting rules to decide whether content fetched by its existing crawlers may enter specified Gemini training or grounding flows (see Google’s common crawlers).

That definition has two parts that should never be collapsed. The token is written with the familiar grammar of robots.txt, but it does not identify an HTTP client. Google explicitly publishes no separate Google-Extended user-agent string.

Google lists the token among its common crawlers because that reference includes product-specific controls as well as request-making crawlers. The directory heading does not override the field-level definition in Google’s current crawler reference.

This is an unusual implementation of the Robots Exclusion Protocol. RFC 9309 normally expects a crawler’s product token to be discoverable in its HTTP user-agent. Google instead documents a special control-capacity model for Google-Extended. The right mental model is therefore policy token, not hidden bot.

2. How a control token without a crawler works

The fetch and the downstream-use decision occur at different layers:

Existing Google user agent fetches a URL
                    │
                    ▼
Google evaluates the host's Google-Extended rules
                    │
          ┌─────────┴─────────┐
          ▼                   ▼
     use permitted       use disallowed
          │                   │
          ▼                   ▼
listed training or      exclude the content from
grounding workflows     those listed workflows

A disallow does not stop the initial request. Googlebot may keep crawling the same URL for Search, and the server will keep recording Googlebot rather than Google-Extended. The policy changes what Google may do with the fetched content in the uses named by its documentation.

That separation also explains why a WAF rule matching Google-Extended is ineffective. There is no request string to match, IP range to attribute to this token, or reverse-DNS identity to verify. Network controls act on the fetch layer; Google-Extended acts after that layer.

Typical training and retrieval bots combine a request identity with a declared purpose. User-triggered agents fetch on a person’s behalf. Google-Extended fits neither pattern cleanly because it expresses permission without sending the request. The broader category differences are mapped in AI crawlers.

Applebot-Extended uses a similar structural split: it does not crawl pages itself and governs data fetched by Applebot. The analogy stops at purpose. Apple’s token is documented for foundation-model training, while Google’s current token also covers specified live grounding.

3. Which Gemini uses it governs

Google’s current description combines two use categories under one token. Calling Google-Extended only a “training opt-out” is now incomplete.

The Google Cloud product names are evolving. The current Grounding with Google Search documentation states that pages disallowing Google-Extended are not used for grounding on Gemini Enterprise Agent Platform. Google has moved older Vertex AI documentation to newer product paths, so policies should track the current canonical reference rather than a frozen product label.

The control’s history is short but materially important:

2023-09-28        2026-06-03          2026-07-22
Launch            Search split        Current verified scope
Bard + Vertex  →  separate Search →   Gemini training + listed
AI model use      AI control test     grounding; Search split below

At launch, Google described a control for improving Bard and Vertex AI generative APIs, including future model generations. Bard later became Gemini, and Google expanded the operational description to name grounding. Treat the affected-product list as maintained vendor policy, not as a permanent statement about every Google AI product.

The safe boundary is literal: apply the token to the uses Google names. Do not expand it to “all Google model training,” every product under Google Gemini, or every use of web content across Google.

4. Google Search, AI Overviews, and AI Mode

The most common misconception is that disallowing Google-Extended removes a site from Google AI Overviews or AI Mode. It does not control whether a site’s links or content can appear in those Search features.

Google’s current controls form three distinct layers:

Search eligibility still starts with Googlebot. Google’s July 2026 optimization guide says a page must be indexed and eligible for a Search snippet, and the property must be included through the Search generative-AI control where that control is available. Search AI uses pages from the Search index for retrieval-augmented generation.

The dedicated Search control is new and not yet universal. In January 2026, Google said it was exploring a separate opt-out. On 3 June, it began testing a Search Console toggle with a subset of UK site owners. As of 22 July, the current help page describes availability more broadly as a rollout to a subset of site owners, not a global release.

Excluding a property through that toggle prevents its links and content from appearing in AI Overviews, AI Mode, and generative AI features in Discover, including as grounding input. The site then receives no impressions or traffic from those features. Google says the choice is not an inclusion or ranking signal for other parts of Search.

The Search Console help page makes the remaining boundary explicit: its toggle does not control AI training. For training of models used to generate Search AI responses, Google points publishers back to Google-Extended. “Google-Extended has no effect on Search ranking” and “it can limit qualifying model training behind Search AI” are therefore both true.

5. Writing the robots.txt rule

To disallow the listed Google-Extended uses across a host, place a named group in that host’s robots.txt file:

User-agent: Google-Extended
Disallow: /

This does not block Googlebot. If Search visibility matters, do not add a Googlebot disallow as a substitute.

A path-scoped policy can keep a public report eligible while excluding the rest of a directory:

User-agent: Google-Extended
Disallow: /members/
Allow: /members/public-report/

The longer, more specific Allow path wins for the public report. Everything else under /members/ remains disallowed for the token’s listed uses.

Google’s robots.txt implementation follows several rules worth testing explicitly:

• Product-token matching is case-insensitive, but URL paths can be case-sensitive.
• Multiple groups matching the same specific token are combined.
• A named token group does not merge with the wildcard * group.
• The most specific matching path wins; an equal-length conflict resolves to the less restrictive rule.

Use the canonical capitalization anyway. It makes review easier and prevents a generic parser, linter, or teammate from mistaking a valid but unusual variant for a typo.

No applicable disallow means the listed use is permitted under this control. An empty Disallow: is not an opt-out, and a comment stating policy has no machine effect.

Robots files are scoped to a scheme, host, and port. A rule served at https://example.com/robots.txt should not be assumed to govern https://docs.example.com, http://example.com, or a service on a non-default port. The protocol details and failure responses are covered in robots.txt; deployment checks belong in the AI crawler access audit.

6. Verifying a policy with no log signature

Verification is configuration-based because Google-Extended produces no independent traffic signal.

Run the check from outside the origin network as well as at the origin. A correct source file can be hidden behind a stale CDN object, a platform-generated robots file, an environment-specific route, or a redirect to the wrong host.

Test at least one allowed path, one disallowed path, and the exact boundary between them. Where several Google-Extended groups exist, test their merged result. Keep a copy of the served response, the change timestamp, the responsible owner, and the policy rationale.

Do not use falling Googlebot volume as a success metric. Googlebot continues to support Search, so stable crawl activity is expected. The AI crawler access audit can verify delivery and generic crawler identities, but its logs cannot reveal a separate Google-Extended bot because none exists.

Finally, schedule a documentation review. The training and grounding scope changed after launch, Google Cloud product paths changed again in 2026, and the separate Search AI control moved from exploration to limited testing within five months.

7. Common misconfigurations

The first error is the most expensive. If the objective is to keep ordinary Search while declining the listed downstream AI uses, leave Googlebot crawlable and address Google-Extended by name. A blanket “block AI” preset should be inspected line by line before deployment.

The security error is the most dangerous. Private, licensed, or customer-only material needs authentication and authorization. A voluntary public file cannot prevent a non-compliant client, a spoofed user agent, or an ordinary browser from retrieving an exposed URL.

8. Choosing an access policy

There is no cost-free setting that rejects only training while preserving every grounding opportunity currently named under Google-Extended. Choose from the actual trade-offs.

The central business choice is training and grounding, not training versus no consequence. A site may reasonably reject future model training yet value being cited as a fresh source in Gemini Apps or a supported Google Cloud grounding product. The current token does not provide separate directives for those objectives.

Search AI adds a second, independent decision. A publisher can keep normal Search while leaving AI Overviews and AI Mode through the limited Search Console control, but that choice sacrifices the traffic and impressions those features could send. It does not replace Google-Extended for qualifying model training.

Record both choices as policy: intended uses, responsible owner, affected properties and hosts, exact directives or Search Console setting, deployment date, and next review date. Use the AI crawler access audit for implementation evidence, then review the platform consequences against Google Gemini and Google AI Overviews.

References

Google product controls and documentation:

• Google Crawling Infrastructure — Google’s common crawlers
• Google Search Console Help — Search generative AI control
• Google Search Central — Optimizing your website for generative AI features on Google Search
• Google Cloud — Grounding with Google Search
• Google — An update on web publisher controls
• Google — Our approach to website controls for Search AI features
• Google — New opportunities, control and insights for website owners

Protocol and comparison:

• Google Crawling Infrastructure — How Google interprets the robots.txt specification
• IETF — RFC 9309: Robots Exclusion Protocol
• Apple — About Applebot