Don't block AI shopping agents at the CDN or WAF

Why AI agents need unblocked access to product pages

When a shopper asks ChatGPT, Claude or Perplexity to compare products, the assistant fetches your pages in real time with a user-initiated agent (ChatGPT-User, Claude-User, Perplexity-User). AI search indexes are built by separate search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot). If your CDN or web application firewall (WAF) answers either group with an error or an interstitial, the assistant has nothing to quote, and your products drop out of the answer.

This happens more often than merchants expect:

• Cloudflare has blocked AI crawlers by default on new zones since July 2025, and its pay per crawl feature answers with HTTP 402.
• Bot-protection vendors (Akamai, DataDome, PerimeterX/HUMAN, AWS WAF) often challenge any non-browser user agent, regardless of what robots.txt says.
• OpenAI and Perplexity document that their user-initiated agents may not follow robots.txt rules (OpenAI, Perplexity), so the WAF is the gate that actually decides.

See AI user agents explained for the difference between training crawlers, search bots and user-initiated agents.

What the scanner checks

The scanner fetches one sampled product page (the homepage if no product page is found) with a regular Chrome user agent, then again with the official user-agent strings of six AI agents and search bots: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User. Training crawlers are not probed.

Each AI response is compared with the browser response:

• Blocked: HTTP 403, 406, 503, 429, 402 (pay per crawl) or any other non-2xx status; or a 200 that is “stripped”, meaning under 30% of the browser page’s size, or missing the product name the browser page shows.
• Challenged: a cf-mitigated: challenge header, an AWS WAF x-amzn-waf-action of challenge or captcha, challenge markup (Cloudflare “Just a moment…”, DataDome, PerimeterX “Press & Hold”, Incapsula, Akamai reference pages), or no response.

Result	Condition
Pass	All six agents get the same page as the browser.
Partial	Some agents are blocked. Scored proportionally (1 − blocked ÷ 6).
Fail	All six agents are blocked.
Inconclusive	Only challenges (no hard blocks), or the page is blocked even for a browser, so agent-specific blocking can’t be isolated.

How to fix AI agent blocking

1. Find the rule that blocks. Check your CDN and WAF logs for the user agents above. Common sources are Cloudflare AI Crawl Control or Bot Fight Mode, Akamai Bot Manager, DataDome, PerimeterX, rate limits keyed on user agent, and app middleware that filters “bot” strings.
2. Allow search bots and user-initiated agents. Keep training crawlers (GPTBot, ClaudeBot) blocked if you want; that is a separate choice and isn’t scored.
3. Verify bots by identity, not by string. Prefer allowlists based on the vendors’ published IP ranges or on Web Bot Auth signatures. ChatGPT agent signs its requests with Signature-Agent: "https://chatgpt.com", and Cloudflare treats signed agents as verified. This lets you admit real agents while still blocking scrapers that fake the user agent.
4. Serve the full page. Don’t return a lighter “bot” template. Allowed agents should get the same server-rendered product name, price and availability a browser gets (see server-rendered product facts).

Verify from a terminal:

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)" \
  https://your-store.com/products/example

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" \
  https://your-store.com/products/example

Both should return 200 with similar byte counts.

How to fix it on Medusa

• Medusa has no built-in bot blocking. The DTC starter storefront is a Next.js app, so whether agents get through depends on your host, CDN and WAF. Check those first, then any storefront middleware you added that filters user agents.
• Any store behind Cloudflare: review AI Crawl Control settings for the zone. A 402 response with a crawler-price header means pay per crawl is charging that agent (Cloudflare docs).
• See Is Medusa agent-ready? for the rest of the checklist.

Common mistakes

• Blocking every user agent that contains “bot”, which catches OAI-SearchBot and PerplexityBot along with scrapers.
• Allowing an agent in robots.txt but challenging it at the WAF. The robots.txt rule is irrelevant if the request never reaches the page.
• Serving a cached interstitial or a “lite” page to non-browser clients, so the agent sees a 200 with no product on it.

Sources

• OpenAI crawlers and user agents
• Anthropic crawlers
• Perplexity crawlers
• Cloudflare: detecting a challenge response
• AWS WAF CAPTCHA and Challenge actions
• Cloudflare signed agents
• OpenAI: ChatGPT agent allowlisting