What's the difference between GPTBot and OAI-SearchBot?

GPTBot crawls content for AI model training. OAI-SearchBot powers ChatGPT Search and Shopping results. You can block GPTBot while allowing OAI-SearchBot. Each respects robots.txt independently.

How often does GPTBot crawl my site?

Not continuously. A 48-day server log study found GPTBot was absent for weeks, then executed 152 requests in a 3-minute burst. It activates when content gains traction in OpenAI's ecosystem.

Should I block or allow GPTBot?

Block GPTBot (training) but allow OAI-SearchBot (search/shopping) and ChatGPT-User (user browsing). This keeps products visible without contributing to model training.

How do I get my products into ChatGPT Shopping?

Allow OAI-SearchBot in robots.txt, implement complete Product + Review + Offer schema in server-rendered JSON-LD, and optionally submit a product feed. ChatGPT Shopping processes 50 million queries daily.

Is Cloudflare blocking AI bots by default?

Since July 2025, Cloudflare defaults to blocking AI bots. Check Cloudflare dashboard Security Bots to verify your settings.

← Back to Blog

How GPTBot Actually Crawls E-commerce Sites — Traffic Data, Server Logs, and What It Means for Your Store

Q: Does GPTBot execute JavaScript?

No. Pre-render.io analyzed 500 million+ GPTBot requests and found zero evidence of JavaScript execution. GPTBot downloads raw HTML only.

April 1, 2026

GPTBot's crawl traffic grew 305% from May 2024 to May 2025, jumping from the #9 crawler to #3 (Cloudflare, 2025). It now accounts for 11.7% of all AI crawler traffic. But GPTBot is just one of three OpenAI crawlers — and each one behaves differently. Understanding which does what is the difference between being visible in ChatGPT and being completely absent.

Three crawlers, three purposes

OpenAI operates three separate user agents, each with different robots.txt behavior (official docs):

Each setting is independent. You can block GPTBot (no training) while allowing OAI-SearchBot (yes to search visibility). This distinction matters: Amazon blocked all three OpenAI crawlers, making 600 million product listings invisible to ChatGPT Shopping (Roketto, 2026).

Recommended robots.txt for e-commerce:

User-agent: GPTBot
Disallow: /       # Block training

User-agent: OAI-SearchBot
Allow: /          # Allow search/shopping

User-agent: ChatGPT-User
Allow: /          # Allow user-triggered browsing

Verify bots by IP, not just user-agent string. OpenAI publishes IP lists at openai.com/gptbot.json, openai.com/searchbot.json, and openai.com/chatgpt-user.json.

GPTBot does not execute JavaScript — and that changes everything

Pre-render.io analyzed over 500 million GPTBot requests and found zero evidence of JavaScript execution (Prerender.io). GPTBot sends an HTTP request, downloads the raw HTML, and moves on. It does not wait for React components to mount, API calls to resolve, or lazy-loaded content to appear.

This is fundamentally different from Googlebot, which uses a headless Chrome rendering engine that executes JavaScript. A React-based Shopify Hydrogen storefront may rank well on Google while being completely blank to GPTBot, ClaudeBot, and PerplexityBot.

Test it yourself: Disable JavaScript in your browser and load your key product pages. If product descriptions, pricing, reviews, or FAQ content disappears — GPTBot can't see it either.

What GPTBot reads from your raw HTML:

What it ignores:

Critical: Your JSON-LD schema must be server-rendered, not JavaScript-injected. If your Product schema is only inserted after JS executes, GPTBot never sees it. Microsoft confirmed Bing uses schema.org markup for Copilot integration. Perplexity does the same. Mismatches between schema and visible HTML trigger deception flags.

Crawl patterns from real server logs

A 48-day server log study (Feb–Mar 2026) revealed GPTBot's actual behavior:

For comparison, the study logged ChatGPT-User at 923 requests (user-triggered), OAI-SearchBot at 330 requests (search), and GPTBot at 187 requests (training) over the same 48-day period.

The crawl-to-referral problem

OpenAI's crawl-to-referral ratio is 1,700:1 — they crawl 1,700 pages for every 1 click they send back to publishers (Cloudflare, June 2025). Anthropic's ratio is 73,000:1. For context, Google's ratio is 14:1.

In the Computer & Electronics vertical specifically, the ratios improve: OpenAI at 401:1 and Perplexity at 88:1 (Cloudflare industry breakdown). E-commerce sites get more referral traffic back from AI crawling than the average.

The Read the Docs project found that blocking AI crawlers decreased their traffic by 75% — from 800GB to 200GB daily — saving approximately $1,500/month in bandwidth costs. For smaller e-commerce sites, the bandwidth impact is lower, but it's worth monitoring.

ChatGPT Shopping: why OAI-SearchBot is the crawler that matters

ChatGPT Shopping processes 50 million shopping queries daily — roughly 2% of ChatGPT's 2.5 billion daily prompts (DataSlayer, 2026). Product recommendations include images, prices, reviews, and direct purchase links — with no ads. OpenAI states recommendations are based on relevance, not paid placement.

For your products to appear in ChatGPT Shopping:

OpenAI also accepts product feeds — structured files (CSV, TSV, XML, or JSON) following their Product Feed Specification. Merchants can submit catalogs with pricing, availability, media assets, and flags like enable_search and enable_checkout.

Shopify merchant onboarding is underway, with early partners including Glossier, SKIMS, Spanx, and Vuori. OpenAI charges a 4% transaction fee on Instant Checkout purchases (on top of standard Stripe processing).

What Cloudflare's data tells us about blocking trends

GPTBot is the most blocked AI crawler — 312 domains block it outright — but also the most explicitly allowed, with 61 domains granting access (Cloudflare). Between July 2025 and January 2026, websites actively blocking AI crawlers outnumbered those blocking Googlebot by 7:1.

Important: Cloudflare defaulted to blocking AI bots in July 2025. Many sites now block GPTBot, ClaudeBot, and PerplexityBot without the site owner knowing. Check your Cloudflare settings under Security → Bots.

GEOlikeaPro's Crawler View simulates exactly what GPTBot, PerplexityBot, and ClaudeBot see when they visit your pages — and what they miss. Join the waitlist to test your store.