AI crawler access report

Can AI crawlers reach halfords.com?

AI crawler access report · checked 2026-11-07 · log in to re-run
88%
halfords.com · AKAMAI-ASN1 - Akamai International B.V., NL
halfords.com blocks 1 of 8 AI crawlers (Claude-User); GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended still get through.
Open to AI crawlersAKAMAI-ASN1 - Akamai International B.V., NL7/8 bots allowed
AI crawlers allowed88%
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Claude-User
PerplexityBot
Google-Extended
Applebot-Extended

Per-bot results

GPTBotrobots.txt allows this bot, no challenge detectedAllowed
OAI-SearchBotrobots.txt allows this bot, no challenge detectedAllowed
ChatGPT-Userrobots.txt allows this bot, no challenge detectedAllowed
ClaudeBotrobots.txt allows this bot, no challenge detectedAllowed
Claude-UserOrigin returned HTTP 403 to the Claude-User user-agent - blocked at the network edgeLikely blocked
PerplexityBotrobots.txt allows this bot, no challenge detectedAllowed
Google-Extendedrobots.txt allows this bot, no challenge detectedAllowed
Applebot-Extendedrobots.txt allows this bot, no challenge detectedAllowed

Origin fingerprint

ASN: AS20940 (AKAMAI-ASN1 - Akamai International B.V., NL)
Bot protection: Cloudflare (passive)

What this means for halfords.com

On halfords.com, 1 AI crawler are blocked - Claude-User, and 7 can reach the site - GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended. The origin resolves to AKAMAI-ASN1 - Akamai International B.V., NL, behind Cloudflare (passive).

Of the blocked crawlers, none are disallowed in robots.txt, and 1 is stopped at the network edge (Claude-User).

halfords.com scores higher than 16% of the 25 sites checked here so far; the average AI access score is 90/100.

Unblock the AI bots on Cloudflare

halfords.com sits behind Cloudflare, so the block is almost certainly the "Block AI bots" control rather than robots.txt. In the Cloudflare dashboard open Security then Bots and turn off "Block AI bots", then check WAF custom rules for anything matching AI user-agents and add an allow rule for the verified bots (Claude-User).

AI crawler access FAQ

Does AI crawler access guarantee halfords.com gets cited?

No. Access is necessary, not sufficient - it removes the blocker. Whether halfords.com gets cited still depends on content, structured data and authority.

How is this measured?

We parse halfords.com's public robots.txt and fingerprint its origin (ASN and bot-protection vendor), then score the share of 8 tracked AI crawlers that can reach it. No private data or IP addresses are shown.

Check your own site free

Run the same per-bot check on any URL, then sign up free to have ChatGPT Search and Perplexity actually fetch it - the real-world test - plus monitoring and exports.

Run a free check