Cloudflare AI Labyrinth, Bot Fight Mode, and AI Crawl Control — Which to Enable
Cloudflare launched three separate AI-focused bot defense features in 2025: Bot Fight Mode (baseline), AI Crawl Control (granular per-crawler policy), and AI Labyrinth (honeypot for misbehaving bots). Each works differently and some combinations will accidentally kill your AI search visibility. This guide explains what each does and the right configuration for e-commerce stores that want to appear in ChatGPT Shopping and Perplexity.
The three features at a glance
| Feature | What it does | Affects AI visibility? |
|---|---|---|
| Bot Fight Mode | Challenges bots from cloud providers & known malicious sources | Can block AI crawlers if misconfigured |
| AI Crawl Control | Per-crawler allow/block with pay-per-crawl options | Direct control — primary tool |
| AI Labyrinth | Honeypot maze for bots that ignore robots.txt | Only affects non-compliant bots |
Bot Fight Mode — the baseline
Bot Fight Mode is Cloudflare’s baseline bot protection, available on all plans including Free. It challenges traffic from known cloud provider IPs and malicious sources.
Stay in the loop
Get news and updates about GEO, AI search and new features. Unsubscribe anytime.
Some legitimate AI crawlers originate from cloud IP ranges (AWS, GCP, Azure). Bot Fight Mode’s blanket challenges can block these even when robots.txt explicitly allows them. Since July 2025, newly created Cloudflare domains have this enabled by default — which is why many merchants discover their store is invisible to ChatGPT without realizing why.
Recommendation: Keep Bot Fight Mode enabled, but use AI Crawl Control (below) to explicitly allow the AI crawlers you want. Don’t rely on Bot Fight Mode alone — it doesn’t distinguish between search crawlers and training crawlers.
AI Crawl Control — the primary tool
AI Crawl Control gives you per-crawler visibility and policy control. You can:
- See which AI crawlers have visited your site and how often
- Set allow/block rules per crawler (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, etc.)
- Request payment from AI crawlers via Cloudflare’s pay-per-crawl marketplace
- Understand crawl patterns to decide which crawlers provide value
Recommended e-commerce configuration
These crawlers power AI search results. Blocking them makes your store invisible to the AI engines your customers use.
- OAI-SearchBot — ChatGPT search index. Powers ChatGPT Shopping citations.
- ChatGPT-User — live user-triggered fetches during real-time queries.
- Claude-SearchBot — Anthropic’s search indexer. Powers Claude web search citations.
- Claude-User — live fetches when Claude users ask about your products.
- PerplexityBot — Perplexity search index. Perplexity converts at 10.5%.
- Google-Extended — Google AI Overviews and Gemini.
These crawlers collect data for training future AI models. They don’t directly drive search traffic. The tradeoff: blocking saves bandwidth and protects content rights, but your data may be underrepresented in future model versions.
- GPTBot — OpenAI model training. Blocking does NOT affect ChatGPT search — OAI-SearchBot handles that separately. A BuzzStream study found 88.2% of sites blocking GPTBot still appear in AI citations.
- ClaudeBot — Anthropic model training. Not the same as Claude-SearchBot (search) or Claude-User (live queries). Blocking ClaudeBot only stops training use (Search Engine Land).
- Bytespider — ByteDance/TikTok crawler. Aggressive crawl volume, minimal referral value for most e-commerce stores.
CCBot powers Common Crawl — the open dataset that trained ChatGPT, Claude, LLaMA, and dozens of other models. Blocking it won’t affect current AI search visibility (live search uses separate crawlers). But your content may be underrepresented in future AI model versions if it’s missing from Common Crawl’s dataset. If bandwidth is a concern (CCBot can consume up to 40% of crawl budget during deep cycles), block it. Otherwise, consider allowing it for long-term AI representation.
The anthropic-ai user agent is deprecated. Anthropic now uses three separate crawlers: ClaudeBot (training), Claude-SearchBot (search indexing), and Claude-User (live queries). If your robots.txt blocks anthropic-ai, it does nothing — update to the current user agent names (Search Engine Journal).
AI Labyrinth — the honeypot
AI Labyrinth is a defensive feature that traps bots which ignore robots.txt. When enabled, Cloudflare injects invisible links on your pages with nofollow and noindex attributes. Legitimate crawlers (that respect these signals) skip the links. Non-compliant bots follow them into a network of AI-generated content that wastes their computational resources.
How it works in practice
- Invisible links added to your pages (not visible to users or compliant crawlers)
- Bots that follow the links enter an AI-generated maze of irrelevant content
- Bots waste resources crawling fake pages instead of your real content
- The system fingerprints bots that go 4+ links deep — no human does that
- Enable with a single toggle in your Cloudflare dashboard — available on all plans including Free
Compliant crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) all respect nofollow and noindex directives — they skip the trap links entirely. Only bots that ignore robots.txt get trapped. Reference: Cloudflare’s AI Labyrinth blog post.
Updated robots.txt for e-commerce (2026)
Based on the current crawler landscape, here’s the recommended robots.txt configuration. Note: Anthropic now uses three separate crawlers — the old anthropic-ai user agent is deprecated.
# === AI SEARCH CRAWLERS (allow — drive citations and traffic) ===
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
# === AI TRAINING CRAWLERS (block — training only, no referral traffic) ===
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Bytespider
Disallow: /
# === CCBot (your call — see notes in article) ===
# Blocking saves bandwidth but may reduce future AI model representation.
# User-agent: CCBot
# Disallow: /Recommended configuration summary
- Bot Fight Mode: ON — baseline protection
- AI Crawl Control: Allow search crawlers (OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended). Block training crawlers (GPTBot, ClaudeBot, Bytespider). Decide on CCBot based on your bandwidth vs. future visibility tradeoff.
- AI Labyrinth: ON — punishes non-compliant bots without affecting legitimate ones
Verify your configuration
- Test robots.txt: Visit
yourstore.com/robots.txt— confirm OAI-SearchBot, Claude-SearchBot, and PerplexityBot are allowed. Confirmanthropic-aiis NOT your only Anthropic directive. - Check AI Crawl Control dashboard: Look for successful crawls from OAI-SearchBot and PerplexityBot in the past 7 days
- Test ChatGPT: Ask “Does [yourstore.com] sell [product]?” — if ChatGPT can fetch real data, the config works
- Test Claude: Ask Claude with web search “what does [yourstore.com] sell?” — verify Claude-SearchBot is indexing
- Test Perplexity: Search for your product category on Perplexity — check if your site appears in citations
Common mistakes
- Blocking all AI traffic: Some tutorials recommend blocking all AI crawlers to “protect content.” This makes your store invisible to ChatGPT Shopping (50M+ daily queries).
- Using deprecated
anthropic-ai: Does nothing. Anthropic now uses ClaudeBot, Claude-SearchBot, and Claude-User. Update your robots.txt. - Leaving Cloudflare defaults untouched: New domains since July 2025 have AI crawlers blocked by default. Explicitly allow the ones you want.
- Confusing training vs. search crawlers: GPTBot ≠ OAI-SearchBot. ClaudeBot ≠ Claude-SearchBot. Blocking the training crawler doesn’t affect search citations.
- Ignoring Perplexity’s stealth crawlers: Even with PerplexityBot blocked, Perplexity has been caught using undeclared crawlers (Cloudflare investigation). AI Labyrinth + WAF rules are needed for actual enforcement.
GEOlikeaPro’s Crawler View shows you exactly which AI crawlers are reaching your pages and which are being blocked at the Cloudflare layer. Sign up free to diagnose your AI search visibility.
FAQ
Does AI Labyrinth hurt my visibility in ChatGPT or Perplexity?
No. AI Labyrinth uses invisible links with nofollow and noindex attributes. Compliant AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) respect these signals and skip the trap entirely. Only bots that ignore robots.txt get caught.
What's the difference between Bot Fight Mode and AI Crawl Control?
Bot Fight Mode is a blanket protection that challenges traffic from cloud IP ranges — it can accidentally block legitimate AI crawlers. AI Crawl Control gives you per-crawler allow/block policy, which is what you need to explicitly allow OAI-SearchBot while blocking training-only bots like GPTBot.
Is anthropic-ai the same as ClaudeBot?
No. anthropic-ai is a deprecated user agent that Anthropic no longer uses. Anthropic now runs three separate crawlers: ClaudeBot (training), Claude-SearchBot (search indexing), and Claude-User (live user queries). Update your robots.txt to use the current names.
Does blocking GPTBot affect ChatGPT search results?
No. GPTBot is for model training only. ChatGPT search uses OAI-SearchBot, a completely separate crawler with different infrastructure and robots.txt directives. A BuzzStream study found 88.2% of sites blocking GPTBot still appear in AI citations.
Should I block CCBot (Common Crawl)?
It depends on your priorities. Blocking CCBot won't affect current AI search visibility (live search uses separate crawlers). But Common Crawl data trains many AI models — blocking it may reduce your representation in future model versions. If bandwidth is a concern (CCBot can consume 40% of crawl budget), block it. Otherwise, consider allowing it.
Why is my new Cloudflare site invisible to ChatGPT?
Since July 2025, newly created Cloudflare domains block AI crawlers by default. Check Security > Bots > AI Crawl Control and explicitly allow OAI-SearchBot, Claude-SearchBot, and PerplexityBot. Also verify that Bot Fight Mode isn't blocking these crawlers at the IP level.