Google-Extended

Google-Extended is a downstream-use control in robots.txt. Unlike an AI crawler, it never visits a site under its own name.

1. What Google-Extended is

Google-Extended is a standalone product token used in a robots.txt User-agent: field. Google applies its rules to decide whether content fetched by existing crawlers may be used for specified Gemini training or grounding purposes (see Google’s common crawlers).

Two details distinguish the token from a crawler. It uses the familiar syntax of robots.txt, but it does not identify an HTTP client. Google explicitly states that Google-Extended has no separate user-agent string.

Although Google lists the token under common crawlers, its current reference defines Google-Extended as a product-specific control rather than a crawler that makes requests.

Google-Extended uses the Robots Exclusion Protocol differently from a typical crawler token. Under RFC 9309, a crawler’s product token is normally discoverable in its HTTP user-agent string. Google documents Google-Extended only as a control token. It functions as a policy token, not as a bot.

2. How a control token without a crawler works

The fetch and the downstream-use decision occur at different layers:

Existing Google user agent fetches a URL
                    │
                    ▼
Google evaluates the host's Google-Extended rules
                    │
          ┌─────────┴─────────┐
          ▼                   ▼
     use permitted       use disallowed
          │                   │
          ▼                   ▼
listed training or      exclude the content from
grounding workflows     those listed workflows

A disallow does not stop the initial request. Googlebot may continue crawling the same URL for Search, and the server will continue recording Googlebot rather than Google-Extended. The policy changes how Google may use the fetched content for the purposes named in its documentation.

This separation also explains why a WAF rule that matches Google-Extended is ineffective. No request carries that string, no IP range belongs specifically to the token, and there is no reverse-DNS identity to verify. Network controls apply during the fetch. Google-Extended governs downstream use afterward.

Typical training and retrieval bots combine a request identity with a declared purpose. User-triggered agents fetch content on a person’s behalf. Google-Extended differs from both because it expresses permission without making the request. AI crawlers compares these categories in more detail.

Applebot-Extended uses a similar arrangement: it does not crawl pages itself and instead governs data fetched by Applebot. The two tokens differ in purpose. Apple documents its token for foundation-model training, while Google’s current token also covers specified live grounding.

3. Which Gemini uses it governs

Google’s current description places two categories of use under the same token. Describing Google-Extended only as a “training opt-out” is therefore incomplete.

Google Cloud product names continue to change. The current Grounding with Google Search documentation states that pages that disallow Google-Extended are not used for grounding on Gemini Enterprise Agent Platform. Google has moved older Vertex AI documentation to newer product paths, so policy reviews should use the current reference rather than rely on an older product name.

The control has a brief history, but its documented scope has changed:

2023-09-28        2026-06-03          2026-07-22
Launch            Separate Search     Documented scope
Bard + Vertex  →  control test     →  Gemini training + listed
AI model use                          grounding; Search controlled separately

At launch, Google described the token as a control over whether content could be used to improve Bard and Vertex AI generative APIs, including future generations of models. Bard later became Gemini, and Google expanded the description to include grounding. The list of affected products reflects Google’s current policy, not a permanent statement about every Google AI product.

Apply the token only to the uses Google names. Its documented scope does not extend to “all Google model training,” every product under Google Gemini, or every use of web content across Google.

4. Google Search, AI Overviews, and AI Mode

A common misconception is that disallowing Google-Extended removes a site from Google AI Overviews or AI Mode. The token does not control whether the site’s links or content can appear in those Search features.

Google’s current controls form three distinct layers:

Search eligibility still begins with Googlebot. Google’s July 2026 optimization guide says that a page must be indexed and eligible for a Search snippet. Where the Search generative AI control is available, the property must also be included under that control. Search AI uses pages from the Search index for retrieval-augmented generation.

The dedicated Search control is new and not yet universally available. In January 2026, Google said it was exploring a separate opt-out. On June 3, it began testing a Search Console toggle with a subset of UK site owners. As of July 22, the current help page describes a broader rollout to a subset of site owners, not a global release.

Excluding a property through that toggle prevents its links and content from appearing in AI Overviews, AI Mode, and generative AI features in Discover, including their use as grounding input. The site then receives no impressions or traffic from those features. Google says the choice is not an inclusion or ranking signal for other parts of Search.

The Search Console help page also states that its toggle does not control AI training. To restrict the training of models used to generate Search AI responses, Google directs publishers to Google-Extended. The token can limit qualifying model training for Search AI without affecting Search rankings.

5. Writing the robots.txt rule

To disallow the specified Google-Extended uses across a host, add a named group to that host’s robots.txt file:

User-agent: Google-Extended
Disallow: /

This rule does not block Googlebot. If Search visibility matters, do not substitute a Googlebot disallow.

A path-specific rule can keep a public report eligible for the specified uses while excluding the rest of its directory:

User-agent: Google-Extended
Disallow: /members/
Allow: /members/public-report/

The longer, more specific Allow path takes precedence for the public report. Everything else under /members/ remains disallowed for the token’s specified uses.

Google’s robots.txt implementation follows several rules worth testing explicitly:

• Product-token matching is case-insensitive, although URL paths can be case-sensitive.
• Multiple groups that match the same specific token are combined.
• A named token group does not merge with the wildcard * group.
• The most specific matching path takes precedence. If conflicting paths have the same length, the less restrictive rule applies.

Use the standard capitalization anyway. It makes review easier and reduces the chance that a generic parser, linter, or teammate will mistake an unusual but valid form for a typo.

If no applicable disallow rule exists, the specified use is permitted under this control. An empty Disallow: is not an opt-out, and a policy written only as a comment has no effect on machines.

Robots files apply to a specific scheme, host, and port. A rule served at https://example.com/robots.txt does not necessarily govern https://docs.example.com, http://example.com, or a service on a non-default port. See robots.txt for protocol details and failure responses, and use the AI crawler access audit for deployment checks.

6. Verifying a policy with no log signature

Because Google-Extended makes no requests under its own identity, verification must focus on the robots.txt configuration.

Check the file from outside the origin network as well as at the origin. A stale CDN object, a platform-generated robots file, an environment-specific route, or a redirect to the wrong host can hide a correct source file.

Test at least one allowed path, one disallowed path, and the exact boundary between them. Where several Google-Extended groups exist, test their merged result. Keep a copy of the served response, the change timestamp, the responsible owner, and the policy rationale.

Do not treat a decline in Googlebot traffic as evidence of success. Googlebot continues to support Search, so stable crawl activity is expected. The AI crawler access audit can verify delivery and identify ordinary crawlers, but server logs cannot reveal a separate Google-Extended bot because none exists.

Schedule a documentation review as well. The training and grounding scope changed after launch, Google Cloud product paths changed again in 2026, and the separate Search AI control moved from exploration to limited testing within five months.

7. Common misconfigurations

If the goal is to keep ordinary Search while declining the specified downstream AI uses, leave Googlebot crawlable and address Google-Extended by name. Review every line of a blanket “block AI” preset before deploying it.

Private, licensed, or customer-only material requires authentication and authorization. A voluntary public file cannot prevent a non-compliant client, a spoofed user agent, or an ordinary browser from retrieving an exposed URL.

8. Choosing an access policy

The current controls do not allow a publisher to reject only training while preserving every grounding opportunity named under Google-Extended. Each available choice carries a trade-off.

Choosing a Google-Extended policy means deciding about both training and grounding. A site may reject future model training while still valuing being cited as a current source in Gemini Apps or a supported Google Cloud grounding product. The current token does not provide separate directives for those two objectives.

Search AI requires a second, independent decision. A publisher can keep ordinary Search while opting out of AI Overviews and AI Mode through the limited Search Console control. That choice gives up the traffic and impressions those features could send, and it does not replace Google-Extended for qualifying model training.

Record both choices in the policy, including the intended uses, the person responsible for the policy, affected properties and hosts, exact directives or Search Console setting, deployment date, and next review date. Use the AI crawler access audit to verify implementation, then review the platform effects in Google Gemini and Google AI Overviews.

References

Google product controls and documentation:

• Google Crawling Infrastructure: Google’s common crawlers
• Google Search Console Help: Search generative AI control
• Google Search Central: Optimizing your website for generative AI features on Google Search
• Google Cloud: Grounding with Google Search
• Google: An update on web publisher controls
• Google: Our approach to website controls for Search AI features
• Google: New opportunities, control and insights for website owners

Protocol and comparison:

• Google Crawling Infrastructure: How Google interprets the robots.txt specification
• IETF: RFC 9309: Robots Exclusion Protocol
• Apple: About Applebot