Amazon Made 600M Products Invisible to ChatGPT. Your Cloudflare Might Be Doing the Same Thing.

Since July 1, 2025, Cloudflare blocks AI crawlers by default on every site. If you haven't touched your bot settings since then, ChatGPT can't see your products. Neither can Perplexity. Neither can Claude.

You're not alone — sites blocking AI crawlers now outnumber those blocking Googlebot by 7 to 1 (Cloudflare). Most of them don't even know it's happening.

Check yours right now — takes 30 seconds

1. Cloudflare Dashboard → select your domain
2. Security → Bots
3. Look for "AI Crawlers" toggle
4. If it says Block — that's the default. Your store is invisible to AI search.

Done? Good. Now here's why this matters and what to do about it.

The numbers that should worry you

Crawler	Purpose	Crawls per referral click
Googlebot	Google Search	14:1
PerplexityBot	Perplexity Search	88:1 (electronics)
OAI-SearchBot	ChatGPT Search	401:1 (electronics)
GPTBot	AI model training	1,700:1
ClaudeBot	AI model training	73,000:1

Source: Cloudflare crawl-to-referral data and industry breakdown

See the difference? PerplexityBot at 88:1 and OAI-SearchBot at 401:1 are search crawlers — they send real customers to your store. GPTBot at 1,700:1 and ClaudeBot at 73,000:1 are training crawlers — they take your content and give almost nothing back.

The default Cloudflare block kills both. You want to block one and allow the other.

The right setup for e-commerce (copy-paste this)

Allow (these send you customers):

• OAI-SearchBot — ChatGPT Search & Shopping
• ChatGPT-User — when users ask ChatGPT to browse your site
• PerplexityBot — Perplexity citations
• ClaudeBot — Claude web search (now on all plans)
• Google-Extended — Google AI Overviews & Gemini

Block (these take your content for training):

• GPTBot — OpenAI model training
• CCBot — Common Crawl
• anthropic-ai — Anthropic model training
• Bytespider — ByteDance

This keeps you visible in ChatGPT Shopping (50M queries/day), Perplexity, Claude, and AI Overviews — while blocking the 80% of AI crawl traffic that's just training and returns zero referral value.

"But what about bandwidth costs?"

Fair question. When Read the Docs blocked all AI crawlers, their bandwidth dropped 75% — from 800 GB/day to 200 GB/day — saving $1,500/month (Search Engine Journal).

But Read the Docs is a documentation site. They don't sell products through AI search. You do. The selective approach above blocks the bandwidth-heavy training crawlers while keeping the search crawlers that actually drive revenue.

“

The Read the Docs project reported that blocking AI crawlers immediately decreased their traffic by 75 percent, going from 800GB per day to 200GB per day. This change saved the project approximately $1,500 per month in bandwidth costs.

One more thing: Perplexity cheats

Even if you block PerplexityBot in robots.txt, Perplexity may still access your content. Cloudflare caught them using stealth, undeclared crawlers that rotate IPs and spoof real-browser user agents (Cloudflare investigation). Cloudflare delisted Perplexity as a verified bot.

If you actually need to block Perplexity, robots.txt alone won't do it. Use Cloudflare's WAF rules or AI Crawl Control.

“

Although Perplexity initially crawls from their declared user agent, when they are presented with a network block, they appear to obscure their crawling identity in an attempt to circumvent the website's preferences.

---

Do this right now

Step 1: Open Cloudflare → Security → Bots. Screenshot your current settings.

Step 2: Switch from blanket block to selective: allow search crawlers, block training crawlers.

Step 3: Check back in 2 weeks. If you're on GEOlikeaPro, run an SOV audit before and after — you'll see the difference in your AI visibility scores.

Join the waitlist — free during beta.