OAI-SearchBot vs GPTBot vs ChatGPT-User — What Each OpenAI Crawler Does

OpenAI runs three separate crawlers: GPTBot, OAI-SearchBot, and ChatGPT-User. Here's the mistake I see constantly: someone configuring robots.txt blocks all three in one line and locks themselves out of ChatGPT Shopping without ever knowing it happened. So let me break down what each crawler actually does, how they're verified, and the config I'd run for e-commerce.

The three crawlers at a glance

Source: OpenAI Crawlers documentation.

GPTBot - the training crawler

GPTBot is OpenAI's crawler for model training. It collects publicly available data to improve future versions of GPT models. User agent: GPTBot/1.1.

Key characteristics:

• Operates offline and asynchronously - not tied to live user activity
• Does not drive referral traffic to your site
• Content scraped by GPTBot may show up in future model responses with no attribution back to you
• Respects robots.txt - blocking it prevents training use

OAI-SearchBot - the search indexer

OAI-SearchBot powers ChatGPT's live search - inline citations, product recommendations, real-time answers. User agent: OAI-SearchBot/1.0.

Key characteristics:

• Asynchronous crawler that builds the ChatGPT search index
• Augments data from Bing and other sources
• Blocking it removes your pages from ChatGPT search results
• Respects robots.txt

ChatGPT-User - the live fetcher

ChatGPT-User fires when a user asks ChatGPT (or a Custom GPT) to fetch a specific URL or browse a site in real time. User agent: ChatGPT-User/1.0.

Key characteristics:

• Not a traditional crawler - behaves more like a browser agent
• Triggered per user query, not on a schedule
• Used when the user explicitly asks for web content
• Respects robots.txt directives targeting ChatGPT-User

Block ChatGPT-User and the user who asks ChatGPT to "check availability on yourstore.com" gets an error instead of your live data. If you want to support agentic shopping workflows at all, keep this one open - that's the whole point of it.

robots.txt configuration for e-commerce

The setup I'd run: block training, allow search and user fetches.

# Block model training
User-agent: GPTBot
Disallow: /

# Allow search indexing (drives ChatGPT Shopping citations)
User-agent: OAI-SearchBot
Allow: /

# Allow live user fetches (supports agentic shopping)
User-agent: ChatGPT-User
Allow: /

That gives you ChatGPT Shopping visibility and supports agent-driven queries, while keeping your content out of future model training without attribution. Best of both, deliberately chosen.

If you want maximum visibility and don't care about the training question (allow all):

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Verification via IP ranges

Each crawler publishes its IP ranges as JSON. Use these to verify legitimate OpenAI traffic in your server logs and to build WAF allow-lists - the user-agent string alone is trivially spoofable, so don't trust it on its own:

• GPTBot IPs: openai.com/gptbot.json
• OAI-SearchBot IPs: openai.com/searchbot.json
• ChatGPT-User IPs: openai.com/chatgpt-user.json

These endpoints change regularly. Automate the fetch and keep your WAF rules current - set it once and forget it and you'll be blocking legitimate OpenAI IPs within a quarter.

Common configuration mistakes

• Blocking all three via User-agent: * Disallow - kills ChatGPT Shopping visibility in one line. Use explicit per-agent directives.
• Cloudflare default blocking (July 2025+) - new Cloudflare domains block AI crawlers by default. Check your Bots dashboard and explicitly allow OAI-SearchBot, because the robots.txt you carefully wrote doesn't matter if Cloudflare returns 403 first.
• Confusing ChatGPT-User with ChatGPT traffic - ChatGPT-User only handles explicit web fetches. Most ChatGPT citations come from OAI-SearchBot's pre-built index, so that's the one to optimize for.
• WAF blocking the user-agent string - robots.txt allows the crawler, your WAF returns 403 anyway. Two layers, check both.

How to check if you're blocked

Quick test: ask ChatGPT "Does [yourstore.com] sell [specific product]?" If it says "I can't access that site" or hands back generic info without visiting, you're blocked at one of these layers:

1. robots.txt disallowing OAI-SearchBot or ChatGPT-User
2. Cloudflare Bot Fight Mode or AI Crawl Control blocking the request
3. WAF rule returning 403 to the OpenAI user agent
4. Server-side rendering failing for the crawler's request headers

Run that test on your own store before you assume you're fine. I've watched store owners swear they were open and fail it on the first try.

GEOlikeaPro's Crawler View shows exactly what each OpenAI crawler sees when it visits your pages - including whether you're blocked at the robots.txt, WAF, or rendering layer. Sign up free to audit your crawler configuration.