OAI-SearchBot vs GPTBot vs ChatGPT-User — What Each OpenAI Crawler Does

OpenAI runs three separate crawlers: GPTBot, OAI-SearchBot, and ChatGPT-User. Practitioners configuring robots.txt often block all three — and unknowingly lock themselves out of ChatGPT Shopping. This guide breaks down what each crawler does, how they’re verified, and the right config for e-commerce.

The three crawlers at a glance

Source: OpenAI Crawlers documentation.

Stay in the loop

Get news and updates about GEO, AI search and new features. Unsubscribe anytime.

Email address Subscribe

GPTBot — the training crawler

GPTBot is OpenAI’s crawler for model training. It collects publicly available data to improve future versions of GPT models. User agent: GPTBot/1.1.

Key characteristics:

• Operates offline and asynchronously — not tied to live user activity
• Does not drive referral traffic to your site
• Content scraped by GPTBot may appear in future model responses without attribution
• Respects robots.txt — blocking it prevents training use

OAI-SearchBot — the search indexer

OAI-SearchBot powers ChatGPT’s live search capabilities — inline citations, product recommendations, and real-time answers. User agent: OAI-SearchBot/1.0.

Key characteristics:

• Asynchronous crawler that builds the ChatGPT search index
• Augments data from Bing and other sources
• Blocking it removes your pages from ChatGPT search results
• Respects robots.txt

ChatGPT-User — the live fetcher

ChatGPT-User is triggered when a user asks ChatGPT (or a Custom GPT) to fetch a specific URL or browse a website in real time. User agent: ChatGPT-User/1.0.

Key characteristics:

• Not a traditional crawler — acts more like a browser agent
• Triggered per user query, not on a schedule
• Used when the user explicitly requests web content
• Respects robots.txt directives targeting ChatGPT-User

Blocking ChatGPT-User means users who ask ChatGPT to “check availability on yourstore.com” will get an error instead of live data. For stores that want to support agentic shopping workflows, keep this allowed.

robots.txt configuration for e-commerce

The recommended setup: block training, allow search and user fetches.

# Block model training
User-agent: GPTBot
Disallow: /

# Allow search indexing (drives ChatGPT Shopping citations)
User-agent: OAI-SearchBot
Allow: /

# Allow live user fetches (supports agentic shopping)
User-agent: ChatGPT-User
Allow: /

This configuration gives you ChatGPT Shopping visibility and supports agent-driven queries while preventing your content from training future models without attribution.

If you want maximum visibility (allow all, including training):

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Verification via IP ranges

Each crawler publishes its IP ranges as JSON. Use these to verify legitimate OpenAI traffic in your server logs and configure WAF allow-lists:

• GPTBot IPs: openai.com/gptbot.json
• OAI-SearchBot IPs: openai.com/searchbot.json
• ChatGPT-User IPs: openai.com/chatgpt-user.json

These endpoints are updated regularly. Automate periodic fetches to keep your WAF rules current.

Common configuration mistakes

• Blocking all three via User-agent: * Disallow — kills ChatGPT Shopping visibility. Use explicit user-agent directives.
• Cloudflare default blocking (July 2025+) — new Cloudflare domains block AI crawlers by default. Check your Bots dashboard and explicitly allow OAI-SearchBot.
• Confusing ChatGPT-User with ChatGPT traffic — ChatGPT-User is only used for explicit web fetches. Most ChatGPT citations come from OAI-SearchBot’s pre-built index.
• WAF blocking the user agent string — robots.txt allows the crawler, but your WAF returns 403. Check both layers.

How to check if you’re blocked

Ask ChatGPT: “Does [yourstore.com] sell [specific product]?” If ChatGPT says “I can’t access that site” or returns generic information without visiting, you’re likely blocked at one of these layers:

1. robots.txt disallowing OAI-SearchBot or ChatGPT-User
2. Cloudflare Bot Fight Mode or AI Crawl Control blocking the request
3. WAF rule returning 403 to the OpenAI user agent
4. Server-side rendering failing for the crawler’s request headers

---

GEOlikeaPro’s Crawler View shows exactly what each OpenAI crawler sees when it visits your pages — including whether you’re blocked at the robots.txt, WAF, or rendering layer. Sign up free to audit your crawler configuration.