How Claude Crawls and Cites Your Store - The 3-Bot Setup
I've covered GPTBot and OpenAI's crawler family in detail, and the most common screwup I see is people blocking one bot to opt out of training and accidentally deleting themselves from the AI's live answers. Anthropic's setup makes that mistake very easy. So here's Claude, bot by bot.
The three crawlers (four, counting Claude Code)
Anthropic runs separate crawlers, and every one of them honors robots.txt independently. That's the key - you can allow and block each by name:
Stay in the loop
Get news and updates about GEO, AI search and new features. Unsubscribe anytime.
Is your brand a Ghost or a Guide on AI?
See if AI knows your brand. We ask Gemini and Claude live - in ~5 seconds, no signup.
"Anthropic updated its crawler documentation this week with a formal breakdown of its three web crawlers and their individual purposes."
| User agent | What it does | Block it and... |
|---|---|---|
ClaudeBot |
Collects public content to train Anthropic's models | Your future content is excluded from training. No effect on live citations. |
Claude-User |
Fetches a page live when a user's question needs it | Claude can't pull your pages in response to user queries. You vanish from answers. |
Claude-SearchBot |
Indexes content to improve Claude's search results | You're not indexed for Claude search. Lower visibility and accuracy in Claude answers. |
claude-code |
The Claude Code CLI fetching on a developer's behalf | Niche - affects dev tooling, not shopping answers. |
People block ClaudeBot to "opt out of AI training" and feel virtuous - fine, that's a legitimate choice. But then they assume they're done. If you actually want to appear in Claude's answers, you must keep Claude-User and Claude-SearchBot allowed. Block all three and you've opted out of the citation, not just the training.
The robots.txt suggested
If your goal is "don't train on me, but do cite me" - which is most brands - here's the shape:
DisallowforClaudeBotonly, if you genuinely want out of trainingAllowforClaude-User- this is what fetches your page when someone asks Claude about your categoryAllowforClaude-SearchBot- this is what gets you into Claude's search index in the first place- Retire the old
anthropic-aidirective - it's deprecated and does nothing now
If you don't care about training and just want maximum visibility, allow all three. The only reason to block ClaudeBot is a deliberate stance on training data.
How Claude decides what to cite
Here's the part that's actually under your control. Claude cites the source that resolves the question - not the page that dances around it with marketing copy. In practice that rewards structure hard. Claude cites the source that resolves the question, and clean schema is how you make yours machine-parseable. Four schema types do the heavy lifting for Claude:
- Article schema with a real author Person entity. Claude weights authorship and expertise. A named author with a bio beats "by Admin" every time.
- FAQPage schema. Question-and-answer pairs are pre-chewed into exactly the shape Claude lifts. See our FAQ schema guide.
- HowTo schema for any step-based content - setup, sizing, install, care.
- Organization schema with consistent NAP. Same name, address, phone everywhere. This is the entity backbone Claude uses to trust you're a real business.
Claude sends lower referral volume than ChatGPT or Perplexity, but the buyers it does send are deliberate and research-heavy. They asked an assistant they trust and acted on its answer. Being the cited brand in that answer is worth more than the raw volume suggests.
Summary
Open your robots.txt today and check it against the table above. Most "we're invisible in Claude" problems are a one-line block doing more than the person who wrote it intended. Fix that, ship the four schema types, and you've done 90% of what Claude optimization actually is.
Want to confirm Claude can read and cite your store as it stands? Run GEOlikeaPro's Visibility Vitals checker - it flags blocked crawlers and missing schema in one pass. And if you haven't read it, our OpenAI crawler breakdown covers the same trap on the ChatGPT side.
FAQ
What's the difference between ClaudeBot, Claude-User and Claude-SearchBot?
ClaudeBot collects public content to train Anthropic's models. Claude-User fetches a page live when a user's question needs it. Claude-SearchBot indexes content to improve Claude's search results. All three honor robots.txt independently, so you control each by name. claude-code is a fourth agent for the Claude Code CLI.
If I block ClaudeBot, do I disappear from Claude's answers?
Not by itself - blocking ClaudeBot only excludes your future content from AI training. But if you also block Claude-User and Claude-SearchBot, you do vanish from Claude's live answers and search index. To stay cited while opting out of training, disallow ClaudeBot only and keep Claude-User and Claude-SearchBot allowed.
Is the old anthropic-ai user agent still valid?
No. The anthropic-ai directive is deprecated and does nothing. Update your robots.txt to the current crawler names - ClaudeBot, Claude-User and Claude-SearchBot - and set allow/disallow rules per bot based on whether you want training, live fetches, and search indexing respectively.
What schema does Claude reward most for citations?
Four types do the heavy lifting: Article schema with a real author Person entity, FAQPage schema, HowTo schema for step-based content, and Organization schema with consistent NAP data. Claude specifically cites the source that most directly resolves the question, and clean structured data is how you make your pages machine-parseable for it.
Latest brands checked with our AI tools
The newest public reports from our AI visibility audit, hosting checker, and AI readiness checker - run yours to add your brand.