Sitemap & IndexNow

1. What sitemap.xml and IndexNow are

Sitemap.xml and IndexNow are two URL-submission protocols with different purposes. Published as an open standard in 2005, sitemap.xml works in pull mode: a publisher hosts a list of URLs at a known location, and search engines fetch it on their own crawl schedules (sitemaps.org). Every major search crawler supports it, including Googlebot, Bingbot, and YandexBot.

Launched by Microsoft and Yandex in 2021, IndexNow works in push mode: a publisher notifies a participating endpoint when a URL changes, and the receiving engine refreshes its index within minutes (indexnow.org). As of 2026-05, the participants are Microsoft Bing, Yandex, Naver, Seznam.cz, and Yep (indexnow.org). Google has not adopted IndexNow since announcing a test in 2021, and no first-party AI crawler participates.

For GEO, both protocols affect AI visibility only through a host search index. Google AI Overviews uses Google’s classic web index, which supports sitemap.xml. Bing Copilot uses Bing’s index, which supports both sitemap.xml and IndexNow. Standalone answer engines such as ChatGPT Search, Perplexity, and Claude run their own retrieval crawlers on their own schedules, and their published crawler documentation does not mention using sitemap.xml or IndexNow.

A page must first be discovered, then fetched by an AI crawler and added to an index, before citability can matter. Submission makes the page a candidate, but it does not make the page citable.

2. How each protocol works

The protocols differ along five operational dimensions and in their scale limits.

An IndexNow submission requires one HTTP request for each URL change and one key file at the site root. It requires neither API tokens nor separate authentication for each engine. A minimal submission looks like this:

# Minimal IndexNow push (illustrative, not a complete client)

1. Host <your-key>.txt at your site root with <your-key> as its contents
   e.g. https://example.com/abc123.txt  →  body: abc123

2. On URL change, GET (single-URL form):
   https://api.indexnow.org/IndexNow?url=https://example.com/page&key=abc123

   Or POST (batch form, up to 10k URLs):
   POST https://<participating-engine>/indexnow
   Content-Type: application/json
   { "host": "example.com",
     "key": "abc123",
     "urlList": ["https://example.com/page1", "https://example.com/page2"] }

3. The receiving engine shares the submission with the other IndexNow
   participants (Bing → Yandex → Naver → Seznam → Yep) automatically.
   Submitting to one reaches all five.

The official specification provides the complete JSON format and the endpoint for each engine. CDNs such as Cloudflare can send IndexNow notifications automatically from the network edge with a single toggle (Cloudflare, 2021), so many sites already send them without adding application code.

Google announced in November 2021 that it would test IndexNow for potential sustainability gains (Search Engine Land, 2021-11-09). The test did not lead to adoption. As of 2026, Google still has not adopted the protocol (ppc.land, 2024-12-30). In practice, one IndexNow push reaches Bing, Yandex, Naver, Seznam, and Yep. Its only path to an AI surface is through Bing Copilot.

3. How each protocol affects AI visibility

The effect of sitemap.xml and IndexNow on an AI surface depends on whether that surface reuses a host search index. The AI surfaces fall into three groups.

Submission protocols affect an AI surface only when that surface uses a search index that supports them. Google AI Overviews gets sitemap support through Google’s index. Bing Copilot gets both sitemap and IndexNow support through Bing’s index. Standalone answer engines use neither protocol through a host search index because they crawl independently. Pushing IndexNow therefore cannot make a page appear in ChatGPT (§6).

4. Which AI submission channels exist

A common question is: How do I submit a URL to ChatGPT, Perplexity, or Claude? As of 2026-05, you cannot submit one directly. No major standalone answer engine offers a first-party URL-submission channel.

Available submission channels: Each reaches an AI surface indirectly through a host search index.

• Google Search Console: Sitemap submission and URL Inspection feed Google’s classic web index, which AI Overviews reuses (Build and submit a sitemap).
• Bing Webmaster Tools: Sitemap submission and URL Submit feed Bing’s index, which Bing Copilot reuses.
• IndexNow: A push notification refreshes the indexes for Bing, Yandex, Naver, Seznam, and Yep within minutes. Through Bing, it reaches Copilot with the lowest available latency (How to add IndexNow).

Unavailable submission channels:

• A first-party URL-submission API from OpenAI, Anthropic, or Perplexity. Their crawler documentation covers robots.txt, IP-range allowlisting, and access policies but does not explain how to submit URLs (OpenAI; Anthropic; Perplexity).
• Google support for IndexNow. Google has not adopted the protocol.
• A single channel that reaches all AI surfaces at once.

Today, submission goes through a host search index rather than directly to an AI engine. Google AI Overviews and Bing Copilot then use those indexes. For ChatGPT or Perplexity, focus on making the page reachable to the retrieval crawler (AI Crawlers) and easy to quote after the crawler reads it (citability).

5. Sitemap.xml, llms.txt, and robots.txt serve different purposes

Three root-level files appear in a complete crawler and discovery setup, and they are often confused. Each serves a distinct purpose.

Sitemap.xml is not a curation file, an access rule, or a “best of” list. It is a complete manifest of the URLs a publisher wants indexed. Reducing it to “only the pages I want AI engines to see” misuses the protocol. Use llms.txt for curation, and keep sitemap.xml complete so that omissions do not create coverage gaps. See the llms.txt §4 comparison for the same distinction.

6. Common submission mistakes

These approaches fail because they confuse the scope of a protocol, its participating engines, or the purpose of a file.

Submission is a prerequisite, but it does not directly improve GEO performance. It makes a page eligible for a host index, which an AI surface may or may not reuse. Selection then depends on other signals, including citability (passage structure and quotable claims), E-E-A-T, and entity recognition. Curating sitemaps or pushing IndexNow to nonparticipating engines will not increase citations. Neglecting submission can prevent the page from becoming a candidate at all.

7. What to do for GEO

Sitemap.xml remains a basic GEO requirement. IndexNow is inexpensive and applies to the AI visibility that comes through Bing Copilot. Neither directly determines whether a page is cited. Those decisions still depend on citability, E-E-A-T, and entity recognition. Configure submission correctly, then focus additional effort on the signals that influence citation.

References

Primary protocol specifications:

• sitemaps.org: Sitemaps XML format — Protocol (an open standard since 2005)
• indexnow.org: Documentation and homepage (participant list)

Primary engine documentation:

• Google Search Central: Build and submit a sitemap
• Microsoft Bing: How to add IndexNow to your website and Bing Webmaster Guidelines
• OpenAI: Overview of OpenAI Crawlers (no submission protocol documented)
• Anthropic: Does Anthropic crawl data from the web (no submission protocol documented)
• Perplexity: Perplexity Crawlers (no submission protocol documented)

Primary historical source:

• Search Engine Land: Google is testing the IndexNow protocol for sustainability (2021-11-09; Google did not adopt the protocol)

Secondary independent coverage and infrastructure:

• ppc.land: Google’s absence from IndexNow raises questions about web indexing standards (2024-12-30; reports Google’s continued non-adoption)
• Cloudflare: Cloudflare now supports IndexNow (2021-10-18; automatic IndexNow notifications through Cloudflare’s edge)
• Search Engine Land: IndexNow — new initiative by Microsoft and Yandex (2021-10-18; launch coverage)