Cloudflare AI Labyrinth, Bot Fight Mode, and AI Crawl Control — Which to Enable
Cloudflare shipped three separate AI bot-defense features in 2025: Bot Fight Mode (baseline), AI Crawl Control (granular per-crawler policy), and AI Labyrinth (a honeypot for misbehaving bots). They work differently, and some combinations will quietly kill your AI search visibility while you think you're protected. Let me explain what each one does and the configuration I'd run for an e-commerce store that actually wants to show up in ChatGPT Shopping and Perplexity.
The three features at a glance
| Feature | What it does | Affects AI visibility? |
|---|---|---|
| Bot Fight Mode | Challenges bots from cloud providers & known malicious sources | Can block AI crawlers if misconfigured |
| AI Crawl Control | Per-crawler allow/block with pay-per-crawl options | Direct control - primary tool |
| AI Labyrinth | Honeypot maze for bots that ignore robots.txt | Only affects non-compliant bots |
Bot Fight Mode - the baseline
Bot Fight Mode is Cloudflare's baseline bot protection, on every plan including Free. It challenges traffic from known cloud-provider IPs and malicious sources.
Stay in the loop
Get news and updates about GEO, AI search and new features. Unsubscribe anytime.
Is your brand a Ghost or a Guide on AI?
See if AI knows your brand. We ask Gemini and Claude live - in ~5 seconds, no signup.
Some legitimate AI crawlers run from cloud IP ranges (AWS, GCP, Azure). Bot Fight Mode's blanket challenges can block those even when your robots.txt explicitly allows them. Since July 2025, newly created Cloudflare domains have this on by default - which is exactly why so many merchants find their store invisible to ChatGPT and never figure out why. They didn't do anything; the default did it for them.
What I'd do: keep Bot Fight Mode enabled, but use AI Crawl Control (below) to explicitly allow the crawlers you want. Don't lean on Bot Fight Mode alone - it doesn't know the difference between a search crawler that sends you customers and a training crawler that doesn't.
AI Crawl Control - the primary tool
AI Crawl Control is where the real control is. Per-crawler visibility and policy. You can:
- See which AI crawlers have hit your site and how often
- Set allow/block rules per crawler (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, etc.)
- Charge AI crawlers via Cloudflare's pay-per-crawl marketplace
- Read crawl patterns to decide which crawlers are actually worth anything to you
The e-commerce configuration I'd run
These crawlers power AI search results. Block them and your store is invisible to the exact AI engines your customers are using to shop.
- OAI-SearchBot - ChatGPT search index. Powers ChatGPT Shopping citations.
- ChatGPT-User - live user-triggered fetches during real-time queries.
- Claude-SearchBot - Anthropic's search indexer. Powers Claude web search citations.
- Claude-User - live fetches when Claude users ask about your products.
- PerplexityBot - Perplexity search index. Perplexity traffic converts at 10.5%.
- Google-Extended - Google AI Overviews and Gemini.
These collect data to train future AI models. They don't drive search traffic. The tradeoff is real but lopsided for e-commerce: blocking saves bandwidth and protects content rights, the cost is your data may be underrepresented in future model versions.
- GPTBot - OpenAI model training. Blocking it does NOT affect ChatGPT search - OAI-SearchBot handles that separately, and people get this backwards constantly. A BuzzStream study found 88.2% of sites blocking GPTBot still appear in AI citations.
- ClaudeBot - Anthropic model training. Not the same as Claude-SearchBot (search) or Claude-User (live queries). Blocking ClaudeBot only stops training use (Search Engine Land).
- Bytespider - ByteDance/TikTok crawler. Aggressive crawl volume, minimal referral value for most e-commerce stores. Easy block.
CCBot powers Common Crawl - the open dataset that trained ChatGPT, Claude, LLaMA, and dozens of other models. Blocking it won't dent your current AI search visibility (live search uses separate crawlers). But your content may be underrepresented in future models if it's missing from Common Crawl's dataset. If bandwidth is the concern (CCBot can eat up to 40% of crawl budget during deep cycles), block it. Otherwise I'd allow it - it's a cheap bet on long-term representation.
The anthropic-ai user agent is deprecated. Anthropic now runs three separate crawlers: ClaudeBot (training), Claude-SearchBot (search indexing), and Claude-User (live queries). If your robots.txt blocks anthropic-ai, it does precisely nothing - update to the current names (Search Engine Journal).
AI Labyrinth - the honeypot
AI Labyrinth is a defensive feature that traps bots which ignore robots.txt. Turn it on and Cloudflare injects invisible links into your pages with nofollow and noindex attributes. Legitimate crawlers respect those signals and skip the links. Non-compliant bots follow them into a maze of AI-generated junk that burns their compute for nothing.
How it works in practice
- Invisible links added to your pages (not visible to users or to compliant crawlers)
- Bots that follow them enter an AI-generated maze of irrelevant content
- Those bots waste resources crawling fake pages instead of your real ones
- The system fingerprints any bot that goes 4+ links deep - no human ever does that
- One toggle in the Cloudflare dashboard, available on all plans including Free
The compliant crawlers you care about (OAI-SearchBot, Claude-SearchBot, PerplexityBot) all respect nofollow and noindex - they skip the trap links entirely. Only the bots ignoring robots.txt get caught. Reference: Cloudflare's AI Labyrinth blog post.
Updated robots.txt for e-commerce (2026)
Based on the current crawler landscape, here's the robots.txt I'd ship. Note the Anthropic change - the old anthropic-ai user agent is deprecated and split into three:
# === AI SEARCH CRAWLERS (allow — drive citations and traffic) ===
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
# === AI TRAINING CRAWLERS (block — training only, no referral traffic) ===
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Bytespider
Disallow: /
# === CCBot (your call — see notes in article) ===
# Blocking saves bandwidth but may reduce future AI model representation.
# User-agent: CCBot
# Disallow: /Recommended configuration summary
- Bot Fight Mode: ON - baseline protection
- AI Crawl Control: allow search crawlers (OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended). Block training crawlers (GPTBot, ClaudeBot, Bytespider). Decide CCBot on your bandwidth-vs-future-visibility tradeoff.
- AI Labyrinth: ON - punishes non-compliant bots without touching the legitimate ones
Verify your configuration
- Test robots.txt: visit
yourstore.com/robots.txt- confirm OAI-SearchBot, Claude-SearchBot, and PerplexityBot are allowed, and thatanthropic-aiis NOT your only Anthropic directive. - Check the AI Crawl Control dashboard: look for successful crawls from OAI-SearchBot and PerplexityBot in the past 7 days
- Test ChatGPT: ask "Does [yourstore.com] sell [product]?" - if it can fetch real data, the config works
- Test Claude: ask Claude with web search "what does [yourstore.com] sell?" - verify Claude-SearchBot is indexing you
- Test Perplexity: search your product category on Perplexity - check whether your site shows up in the citations
Common mistakes
- Blocking all AI traffic: plenty of tutorials still tell you to block every AI crawler to "protect content." That makes your store invisible to ChatGPT Shopping (50M+ daily queries). It's the most expensive piece of advice on the internet right now.
- Using the deprecated
anthropic-ai: does nothing. Anthropic uses ClaudeBot, Claude-SearchBot, and Claude-User now. Update your robots.txt. - Leaving Cloudflare defaults untouched: new domains since July 2025 block AI crawlers by default. Explicitly allow the ones you want - silence is a block here, not a neutral.
- Confusing training vs search crawlers: GPTBot ≠ OAI-SearchBot. ClaudeBot ≠ Claude-SearchBot. Blocking the training crawler does not touch your search citations.
- Ignoring Perplexity's stealth crawlers: even with PerplexityBot blocked, Perplexity has been caught using undeclared crawlers (Cloudflare investigation). robots.txt alone is theater - AI Labyrinth plus WAF rules is what actual enforcement looks like.
GEOlikeaPro's Crawler View shows exactly which AI crawlers are reaching your pages and which are being blocked at the Cloudflare layer. Sign up free to diagnose your AI search visibility.
FAQ
Does AI Labyrinth hurt my visibility in ChatGPT or Perplexity?
No. AI Labyrinth uses invisible links with nofollow and noindex attributes. Compliant AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) respect these signals and skip the trap entirely. Only bots that ignore robots.txt get caught.
What's the difference between Bot Fight Mode and AI Crawl Control?
Bot Fight Mode is a blanket protection that challenges traffic from cloud IP ranges — it can accidentally block legitimate AI crawlers. AI Crawl Control gives you per-crawler allow/block policy, which is what you need to explicitly allow OAI-SearchBot while blocking training-only bots like GPTBot.
Is anthropic-ai the same as ClaudeBot?
No. anthropic-ai is a deprecated user agent that Anthropic no longer uses. Anthropic now runs three separate crawlers: ClaudeBot (training), Claude-SearchBot (search indexing), and Claude-User (live user queries). Update your robots.txt to use the current names.
Does blocking GPTBot affect ChatGPT search results?
No. GPTBot is for model training only. ChatGPT search uses OAI-SearchBot, a completely separate crawler with different infrastructure and robots.txt directives. A BuzzStream study found 88.2% of sites blocking GPTBot still appear in AI citations.
Should I block CCBot (Common Crawl)?
It depends on your priorities. Blocking CCBot won't affect current AI search visibility (live search uses separate crawlers). But Common Crawl data trains many AI models — blocking it may reduce your representation in future model versions. If bandwidth is a concern (CCBot can consume 40% of crawl budget), block it. Otherwise, consider allowing it.
Why is my new Cloudflare site invisible to ChatGPT?
Since July 2025, newly created Cloudflare domains block AI crawlers by default. Check Security > Bots > AI Crawl Control and explicitly allow OAI-SearchBot, Claude-SearchBot, and PerplexityBot. Also verify that Bot Fight Mode isn't blocking these crawlers at the IP level.