Site Navigation for AI Crawlability — Training Crawlers and Retrieval Crawlers Need Different Things

Almost every "AI crawlability" guide treats AI crawlers as a single audience and gives one set of advice: flatten your site, fix breadcrumbs, kill JS nav. That advice is half right and half actively harmful, because there are two categories of AI crawler hitting your store and they want opposite things from your navigation. Optimize for one and you punish the other.

“

AI crawlers serve two fundamentally different purposes: they either train models (by extracting your content to improve AI capabilities) or retrieve content for generative answers (by pulling your information to answer user questions directly within AI interfaces).

The two-category map

The user-agent string tells you which mode the crawler is in. The mode tells you which of your navigation choices it actually uses.

Stay in the loop

Get news and updates about GEO, AI search and new features. Unsubscribe anytime.

Email address Subscribe

The split is documented at the source: OpenAI's bot directory separates GPTBot (training) from OAI-SearchBot and ChatGPT-User (retrieval), and Anthropic publishes the same split for ClaudeBot vs Claude-User. Cloudflare's 2025 crawler study shows the two categories driving very different traffic patterns across its network. For more on the OpenAI side specifically, see our crawler comparison and the GPTBot traffic data piece.

Click depth: a training-crawler concept, not a retrieval one

Pages four or more clicks from the homepage are statistically less likely to be discovered by a training crawler walking your link graph. That is a real effect, and it's the only piece of "flatten your nav" advice that survives scrutiny.

“

Distance From Index (DFI) shows how many clicks separate a page from the homepage. It remains one of the strongest signals influencing how both search crawlers and AI bots prioritize your content.

It does not apply to retrieval crawlers at all. When a shopper asks ChatGPT "which Allbirds Tree Runner is on sale," OAI-SearchBot is handed a URL that's already three layers deep into your collection structure. It doesn't traverse from the homepage — it fetches the deep URL directly. Click depth is irrelevant to that fetch.

Practical translation:

• Keep canonical product URLs reachable in 2–3 clicks from the homepage so training crawlers find them at all. This is what determines whether a product enters the corpus that next year's models are trained on.
• Stop optimizing variant URLs and filtered combinations for crawl depth. They're for retrieval crawlers, which arrive directly. Don't link them from the homepage just to flatten depth — you'll dilute your link graph for training and gain nothing.

Anchor IDs: the missing layer for retrieval crawlers

Retrieval crawlers don't cite URLs. They cite chunks. Open any Perplexity answer and inspect the citation links — they end in #:~:text=... (a text fragment, the W3C-standardized URL form for deep-linking arbitrary text) or #section-id (a named anchor). ChatGPT's inline citations and Google AI Overviews do the same. The model wants to point its user at the exact paragraph that supports the claim, not the page top.

For that to work, your H2/H3/H4 tags need stable, descriptive id attributes. Most Shopify themes, blog templates, and CMS exports omit them entirely — which means a retrieval crawler can index the page, but can't deep-cite anything inside it, so it picks a different source that can.

<!-- Bad: invisible to chunk-level citation -->
<h2>Sizing guide</h2>

<!-- Good: addressable, stable, linkable -->
<h2 id="sizing-guide">Sizing guide</h2>

<!-- Better: paragraph-level for long-form -->
<p id="return-window">Returns are accepted within 30 days...</p>

The same principle applies to FAQ answers. Each <dt>/<summary> block should carry an id derived from the question slug. That's the difference between "the model knew your store had a return policy" and "the model quoted your return policy verbatim and linked the exact line."

llms.txt: a navigation file most stores still don't ship

Sitemap.xml was built for traditional search crawlers — a flat list of every URL with last-modified timestamps. It's a poor fit for AI agents, which want a small, prioritized index of the documents that actually answer questions. The community-proposed answer is /llms.txt (and the longer /llms-full.txt), a markdown file at the site root that lists your highest-value pages with one-sentence descriptions. The full proposal lives at llmstxt.org.

# Acme Running

Specialty running shoe retailer. Independent sizing and gait advice.

## Products
- [CloudRunner Pro](https://acme.com/products/cloudrunner-pro): Daily trainer, 4mm drop, 280g
- [TrailMax 7](https://acme.com/products/trailmax-7): Aggressive lugged trail shoe, waterproof

## Policies
- [Returns](https://acme.com/policies/returns): 30-day window, free return shipping
- [Sizing guide](https://acme.com/pages/sizing): How our sizing compares to Nike, Adidas, Hoka

Two things to keep in mind. First, llms.txt is not yet a ratified standard, so don't expect it to do heavy lifting; treat it as cheap insurance rather than a primary channel. Second, ship /llms.txt at the apex domain (not /blog/llms.txt) and link it from the HTML head with <link rel="alternate" type="text/markdown" href="/llms.txt"> so retrieval crawlers can discover it without guessing the path.

HTTP Link headers: a markdown side door

Retrieval crawlers parse HTML, but they parse markdown faster and more reliably (no boilerplate stripping, no nav-bar deduplication). You can serve a markdown rendering of any page alongside the HTML using HTTP Link headers, with no impact on what humans see:

HTTP/2 200 OK
Content-Type: text/html
Link: </products/cloudrunner-pro.md>; rel="alternate"; type="text/markdown"
Link: <https://acme.com/products/cloudrunner-pro>; rel="canonical"

The crawler that prefers markdown follows the alternate; the crawler that doesn't ignores the header and reads the HTML. ChatGPT's Atlas browser and several agentic frameworks already prefer markdown alternates when offered. The build cost is small — most static-site generators emit markdown anyway, and Shopify/Next.js/Astro stores can render .md via the same product loader as HTML.

Faceted navigation: the advice you've heard is wrong for half your audience

The standard advice — "block ?color= in robots.txt, canonicalize filtered URLs to the unfiltered collection" — is correct for training crawlers and harmful for retrieval crawlers. A filtered collection URL is exactly what a retrieval crawler arrives at when a user asks "red running shoes size 10 under $150." Block it and you remove yourself from that query. Google's canonicalization guidance covers the mechanics; the part the guides skip is that the right rule depends on whether the filter is decorative or semantic.

The right configuration distinguishes by intent, not by URL pattern:

1. Use noindex, follow on filtered pages, not robots.txt blocks. Training crawlers see "don't add this to the corpus, but the links inside are fine to walk." Retrieval crawlers see a perfectly servable page when their user requests it directly.
2. Reserve robots.txt blocks for true infinite spaces: session IDs, sort-order combos, paginated facets beyond page 50. Anything a human would never deep-link to.
3. Set rel="canonical" from filtered to unfiltered only when the filter is decorative (sort order, view mode). Keep the canonical self-referential when the filter is semantic (color, size, price band) — the retrieval crawler's user asked for that filter.
4. Don't nofollow internal facet links. That's a 2014 SEO reflex. It hides paths from training crawlers without doing anything useful.

Diagnose from server logs, not from theory

Stop guessing. Your access logs already tell you whether each crawler category is reaching what you want. The diagnostic is a one-liner you can run on any nginx/Cloudflare log export:

# What URLs is each crawler hitting?
awk -F'"' '/GPTBot|ClaudeBot|CCBot/      {print "TRAIN  " $2}
            /OAI-SearchBot|PerplexityBot|ChatGPT-User|Claude-User/ {print "RETR   " $2}' \
  access.log | sort | uniq -c | sort -rn | head -50

Read the output against this checklist:

• Training crawlers hitting only your homepage and sitemap.xml. Your link graph is broken — they can't traverse beyond the entry. Check JS-only navigation and orphan collections.
• Retrieval crawlers hitting deep filtered URLs you've been blocking. Your robots.txt is killing your AI search visibility. Switch those URLs to noindex, follow.
• Retrieval crawlers fetching #anchor URLs that 200 but render the wrong section. Your headings have id attributes but they don't match what the crawler is requesting — the model is hallucinating fragment IDs because there's no canonical list. Ship llms.txt with stable anchors.
• Training crawlers stuck on faceted URLs (10x your product count). Standard facet bloat — apply the noindex, follow + selective canonical fix above.
• Zero hits from one category. Something is hard-blocking the user-agent. Check WAF, Cloudflare bot rules, and robots.txt. See Cloudflare's default AI-crawler blocks for the most common cause.

The actual ship list

If you take only one pass at navigation for AI crawlers, do these in order. Each one is cheap, and each one closes a gap that exists on roughly nine out of ten Shopify and DTC stores we audit.

1. Add semantic id attributes to every H2/H3 in product, collection, policy, and FAQ templates. One template change, full-catalog impact.
2. Ship /llms.txt at the apex domain. Five high-value pages with one-sentence descriptions is enough to start.
3. Replace robots.txt blocks on filter URL patterns with <meta name="robots" content="noindex, follow">. Keep blocks only for true infinite spaces.
4. Verify breadcrumbs render server-side and emit BreadcrumbList JSON-LD. This is the one piece of the old "flatten your nav" advice that still earns its keep, because it helps both crawler categories.
5. Run the awk one-liner above against this week's access logs. Decide what to fix based on what crawlers are actually doing, not what guides assume they do.

Sources

• OpenAI — Overview of OpenAI Crawlers (GPTBot, OAI-SearchBot, ChatGPT-User user-agent strings, IP ranges, opt-out)
• Anthropic — ClaudeBot & Claude-User crawler policy
• Cloudflare — From Googlebot to GPTBot: who's crawling your site in 2025
• llmstxt.org — the /llms.txt proposal (Jeremy Howard et al.)
• MDN — URL Text Fragments (the #:~:text= spec retrieval crawlers cite into)
• Google Search Central — Canonicalization

---

GEOlikeaPro's Crawler View separates training-mode and retrieval-mode hits, surfaces the URLs each category is reaching (and the ones it isn't), and flags missing anchor IDs and llms.txt at the template level. Sign up free to run the audit on your store.