Sitemap & IndexNow

1. What sitemap.xml and IndexNow are

Sitemap.xml and IndexNow are two URL-submission protocols that solve adjacent but distinct problems. Sitemap.xml is the older one — published in 2005 as an open standard (sitemaps.org) — and works in pull mode: a publisher hosts a list of URLs at a known path, and search engines fetch the file on their own crawl schedule. Every major search crawler honors it, including Googlebot, Bingbot, and YandexBot.

IndexNow is the newer one — launched in 2021 by Microsoft and Yandex (indexnow.org) — and works in push mode: a publisher notifies a participating endpoint the moment a URL changes, and the receiving engine refreshes its index in minutes. The participant list as of 2026-05 is Microsoft Bing, Yandex, Naver, Seznam.cz, and Yep (indexnow.org). Google has not adopted IndexNow since its 2021 testing announcement, and no AI vendor’s first-party crawler participates.

For GEO purposes, the load-bearing fact is downstream of either protocol’s mechanics: their AI-search effect transits only through the host search index. Google AI Overviews inherits Google’s sitemap behavior because AIO grounds in Google’s classic web index; Bing Copilot inherits Bing’s sitemap and IndexNow behavior for the same reason. Standalone answer engines — ChatGPT Search, Perplexity, Claude — run their own retrieval crawlers on their own crawl schedules, and their published crawler documentation does not mention sitemap.xml or IndexNow consumption.

In the answer-loop sequence, this layer sits upstream of AI Crawlers: discovery (you exist on someone’s URL list) comes before crawl (a bot fetches you), which comes before retrievability (the bot’s index includes you), which comes before citability (you are liftable when read). Submission is necessary, not sufficient — clearing it makes you a candidate, not a citation.

2. The mechanics — what each protocol actually does

The two protocols differ on five operational dimensions, summarized below.

The publisher-side IndexNow surface is intentionally minimal — one HTTP request per URL change, one key file at the site root, no API tokens, no per-engine authentication. A complete minimal push looks like:

# IndexNow — the minimal push, the principle not a full client

1. Host <your-key>.txt at your site root, body containing <your-key>
   e.g. https://example.com/abc123.txt  →  body: abc123

2. On URL change, GET (single-URL form):
   https://api.indexnow.org/IndexNow?url=https://example.com/page&key=abc123

   Or POST (batch form, up to 10k URLs):
   POST https://<participating-engine>/indexnow
   Content-Type: application/json
   { "host": "example.com",
     "key": "abc123",
     "urlList": ["https://example.com/page1", "https://example.com/page2"] }

3. The receiving engine shares the submission with the other IndexNow
   participants (Bing → Yandex → Naver → Seznam → Yep) automatically.
   Submit to one, reach all five.

The full JSON shape and per-engine endpoint list are in the official spec; CDNs including Cloudflare publish IndexNow at the edge with a single toggle (Cloudflare, 2021), so many sites already emit IndexNow signals without site-side code.

A note on Google and IndexNow: Google announced in November 2021 that it would test the protocol for sustainability gains (Search Engine Land, 2021-11-09). The test did not move to adoption; as of 2026 independent coverage continues to treat Google’s absence as the status quo (ppc.land, 2024-12-30). The operational implication: a single IndexNow push reaches Bing, Yandex, Naver, Seznam, and Yep — and through them, on the AI side, only Bing Copilot.

3. Per-engine effect on AI visibility — the canonical table

The single most-quoted asset on this page. Sitemap.xml and IndexNow’s effect on an AI surface is determined by whether that surface reuses a host search index, and the engines split cleanly into three groups.

The load-bearing read of this table: submission protocols affect an AI surface only where that surface inherits a search ranking that uses them. Google AI Overviews inherits, Bing Copilot inherits twice (sitemap + IndexNow), and standalone answer engines inherit nothing — they crawl on their own. “Push IndexNow to appear in ChatGPT” is therefore a category error (§6).

4. “Submit to the AI index” — what does and does not exist

The most common practitioner question on this layer is: how do I submit my URL to ChatGPT, Perplexity, or Claude? As of 2026-05, the answer is: directly, you can’t. No first-party URL-submission channel exists for any major standalone answer engine.

Submission channels that do exist — each reaches an AI surface transitively, via a host search index:

• Google Search Console — sitemap submission and URL Inspection feed Google’s classic web index, which AI Overviews reuses (Build and submit a sitemap)
• Bing Webmaster Tools — sitemap submission and URL Submit feed Bing’s index, which Bing Copilot reuses
• IndexNow — push notification refreshes Bing’s index (and Yandex’s, Naver’s, Seznam’s, Yep’s) in minutes; reaches Bing Copilot transitively at the lowest available latency (How to add IndexNow)

Submission channels that do not exist:

• A first-party OpenAI / Anthropic / Perplexity URL-submission API — their crawler docs cover robots.txt, IP-range allowlisting, and access policy but say nothing about how to submit URLs (OpenAI; Anthropic; Perplexity)
• IndexNow consumption by Google — frequently asked, but Google has not adopted the protocol
• Any single channel that reaches all AI surfaces at once

The honest framing: there is no “submit to AI” today — only “submit to a host index” and wait for the AI surface to inherit. When a client asks where to push their page so it shows up in ChatGPT or Perplexity, the answer is not a submission protocol. It is the work that makes the page liftable once a retrieval crawler reaches it (citability) plus making sure that crawler can reach the page at all (AI Crawlers).

5. Sitemap.xml ≠ llms.txt ≠ robots.txt — three files, three jobs

Three root-level files appear in any complete crawler-and-discovery setup, and they are routinely confused. Each one does exactly one job and refuses to do the others.

The load-bearing line: sitemap.xml is not a curation file, not an access rule, not a “best of” list. It is a completeness manifest. Trying to slim sitemap.xml down to “only the pages I want AI engines to see” misreads the protocol’s job — curation belongs to llms.txt; sitemap.xml must reflect everything you want indexed, or the engine treats the omission as a coverage gap. The same direction inverted gives the llms.txt §4 three-file table — read together they are the canonical sitemap-vs-llmstxt disambiguation, mirrored from each protocol’s own side.

6. Anti-patterns — when submission backfires or wastes effort

Each anti-pattern below sounds right and fails because it confuses a protocol’s scope, a participating-engine list, or a file’s job.

The line that closes the section: submission is a hygiene baseline, not a GEO lever. It earns you candidacy at the host index, which an AI surface may or may not reuse. The signals that decide whether you are actually picked once you are in the candidate pool live elsewhere — citability (chunk shape, quotable claims), E-E-A-T, entity recognition. Over-investing in the submission layer (curating sitemaps, pushing IndexNow at non-participating engines) does not move citation; under-investing forfeits candidacy. Get it correct and move on.

7. Why this matters for GEO + how to act

The closing read, stated plainly: sitemap.xml is a hygiene baseline GEO never drops; IndexNow is a low-cost investment worth making for the share of your AI exposure that flows through Bing Copilot. Neither is a GEO load-bearing lever — those still live in citability, E-E-A-T, and entity recognition. Get the indexing layer correct, then spend marginal effort where citation is actually decided.

References

Primary — protocol specifications:

• sitemaps.org — Sitemaps XML format — Protocol (open standard, since 2005)
• indexnow.org — Documentation · Homepage (participant list)

Primary — engine documentation:

• Google Search Central — Build and submit a sitemap
• Microsoft Bing — How to add IndexNow to your website · Bing Webmaster Guidelines
• OpenAI — Overview of OpenAI Crawlers (no submission protocol documented)
• Anthropic — Does Anthropic crawl data from the web (no submission protocol documented)
• Perplexity — Perplexity Crawlers (no submission protocol documented)

Primary — historical anchor:

• Search Engine Land — Google is testing the IndexNow protocol for sustainability (2021-11-09; the announcement that did not move to adoption)

Secondary — independent coverage and infrastructure:

• ppc.land — Google’s absence from IndexNow raises questions about web indexing standards (2024-12-30; current-status framing)
• Cloudflare — Cloudflare now supports IndexNow (2021-10-18; edge-level automatic IndexNow push)
• Search Engine Land — IndexNow — new initiative by Microsoft and Yandex (2021-10-18; launch coverage)