# Don't block AI shopping agents at the CDN or WAF

> An agent-friendly store serves AI search bots and user-initiated AI agents (ChatGPT-User, Claude-User, Perplexity-User) the same product page a browser gets. WAF rules, bot-protection products and default CDN settings often return a 403, a 402 or a challenge instead, which makes the store invisible to AI shopping answers.

- Source: https://is-agentic-storefront.com/checks/agent-access
- Updated: 2026-10-15
- Category: Access (25% of score)
- Impact: High
- Spec: [OpenAI crawler docs](https://developers.openai.com/api/docs/bots)
- Scanner check: AI agents and AI search bots are not blocked

## Why AI agents need unblocked access to product pages

When a shopper asks ChatGPT, Claude or Perplexity to compare products, the assistant fetches your pages in real time with a user-initiated agent (`ChatGPT-User`, `Claude-User`, `Perplexity-User`). AI search indexes are built by separate search bots (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`). If your CDN or web application firewall (WAF) answers either group with an error or an interstitial, the assistant has nothing to quote, and your products drop out of the answer.

This happens more often than merchants expect:

- Cloudflare has [blocked AI crawlers by default on new zones since July 2025](https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/), and its [pay per crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages/) feature answers with HTTP 402.
- Bot-protection vendors (Akamai, DataDome, PerimeterX/HUMAN, AWS WAF) often challenge any non-browser user agent, regardless of what [robots.txt](https://is-agentic-storefront.com/checks/robots-txt) says.
- OpenAI and Perplexity document that their user-initiated agents may not follow robots.txt rules ([OpenAI](https://developers.openai.com/api/docs/bots), [Perplexity](https://docs.perplexity.ai/guides/bots)), so the WAF is the gate that actually decides.

See [AI user agents explained](https://is-agentic-storefront.com/guides/ai-user-agents) for the difference between training crawlers, search bots and user-initiated agents.

## What the scanner checks

The scanner fetches one sampled product page (the homepage if no product page is found) with a regular Chrome user agent, then again with the official user-agent strings of six AI agents and search bots: `OAI-SearchBot`, `ChatGPT-User`, `Claude-SearchBot`, `Claude-User`, `PerplexityBot` and `Perplexity-User`. Training crawlers are not probed.

Each AI response is compared with the browser response:

- **Blocked**: HTTP 403, 406, 503, 429, 402 (pay per crawl) or any other non-2xx status; or a 200 that is "stripped", meaning under 30% of the browser page's size, or missing the product name the browser page shows.
- **Challenged**: a `cf-mitigated: challenge` header, an AWS WAF `x-amzn-waf-action` of `challenge` or `captcha`, challenge markup (Cloudflare "Just a moment...", DataDome, PerimeterX "Press & Hold", Incapsula, Akamai reference pages), or no response.

| Result       | Condition                                                                                                                  |
| ------------ | -------------------------------------------------------------------------------------------------------------------------- |
| Pass         | All six agents get the same page as the browser.                                                                           |
| Partial      | Some agents are blocked. Scored proportionally (1 − blocked ÷ 6).                                                          |
| Fail         | All six agents are blocked.                                                                                                |
| Inconclusive | Only challenges (no hard blocks), or the page is blocked even for a browser, so agent-specific blocking can't be isolated. |

## How to fix AI agent blocking

1. **Find the rule that blocks.** Check your CDN and WAF logs for the user agents above. Common sources are Cloudflare AI Crawl Control or Bot Fight Mode, Akamai Bot Manager, DataDome, PerimeterX, rate limits keyed on user agent, and app middleware that filters "bot" strings.
2. **Allow search bots and user-initiated agents.** Keep training crawlers (`GPTBot`, `ClaudeBot`) blocked if you want; that is a separate choice and isn't scored.
3. **Verify bots by identity, not by string.** Prefer allowlists based on the vendors' published IP ranges or on [Web Bot Auth](https://datatracker.ietf.org/doc/html/draft-meunier-web-bot-auth-architecture) signatures. ChatGPT agent signs its requests with `Signature-Agent: "https://chatgpt.com"`, and Cloudflare treats [signed agents as verified](https://developers.cloudflare.com/bots/concepts/bot/signed-agents/). This lets you admit real agents while still blocking scrapers that fake the user agent.
4. **Serve the full page.** Don't return a lighter "bot" template. Allowed agents should get the same server-rendered product name, price and availability a browser gets (see [server-rendered product facts](https://is-agentic-storefront.com/checks/ssr-product-facts)).

Verify from a terminal:

```bash
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)" \
  https://your-store.com/products/example

curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36" \
  https://your-store.com/products/example
```

Both should return `200` with similar byte counts.

## How to fix it on Medusa

- **Medusa has no built-in bot blocking.** The [DTC starter](https://github.com/medusajs/dtc-starter) storefront is a Next.js app, so whether agents get through depends on your host, CDN and WAF. Check those first, then any storefront middleware you added that filters user agents.
- **Any store behind Cloudflare**: review AI Crawl Control settings for the zone. A 402 response with a `crawler-price` header means pay per crawl is charging that agent ([Cloudflare docs](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages/)).
- See [Is Medusa agent-ready?](https://is-agentic-storefront.com/guides/medusa-agent-readiness) for the rest of the checklist.

## Common mistakes

- Blocking every user agent that contains "bot", which catches `OAI-SearchBot` and `PerplexityBot` along with scrapers.
- Allowing an agent in robots.txt but challenging it at the WAF. The robots.txt rule is irrelevant if the request never reaches the page.
- Serving a cached interstitial or a "lite" page to non-browser clients, so the agent sees a 200 with no product on it.

## Sources

- [OpenAI crawlers and user agents](https://developers.openai.com/api/docs/bots)
- [Anthropic crawlers](https://support.claude.com/en/articles/8896518)
- [Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
- [Cloudflare: detecting a challenge response](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/detect-response/)
- [AWS WAF CAPTCHA and Challenge actions](https://docs.aws.amazon.com/waf/latest/developerguide/waf-captcha-and-challenge-actions.html)
- [Cloudflare signed agents](https://developers.cloudflare.com/bots/concepts/bot/signed-agents/)
- [OpenAI: ChatGPT agent allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-works-cloud-browser-allowlisting)

## Frequently asked questions

### Does blocking GPTBot hurt my agent-readiness score?

No. GPTBot, ClaudeBot and other training crawlers are not probed by this check. Blocking training is a business decision; blocking the search bots and user-initiated agents that shop on a person's behalf is what costs you sales.

### Why is my result inconclusive instead of fail?

The scanner sends spoofed user-agent strings from its own IP. A bot challenge on those requests may not reflect what verified bots see, because bot-management products check IP ranges and signatures. Challenges alone are reported as inconclusive and excluded from the score.

### How do I test whether ChatGPT can read my product page?

Run curl -A "ChatGPT-User/1.0" against a product URL and confirm you get HTTP 200 with the product name and price in the HTML. Repeat with Claude-User and Perplexity-User.

## Run your store on a platform built for agentic commerce

Medusa is open-source, headless commerce with a full Store API agents can call and storefronts you control end to end.

### Deploy on Medusa Cloud

Open-source commerce with a full Store API, hosted and scaled for you. Your storefront, your data model.

- [Deploy on Medusa Cloud](https://cloud.medusajs.com?utm_source=agent-friendly-scorer&utm_medium=referral&utm_campaign=markdown-cta&utm_content=%2Fchecks%2Fagent-access)
- [Medusa Cloud docs](https://docs.medusajs.com/cloud?utm_source=agent-friendly-scorer&utm_medium=referral&utm_campaign=markdown-cta&utm_content=%2Fchecks%2Fagent-access)

### Storefront skills for coding agents

Teach Claude Code, Cursor or Codex to build storefronts with the patterns these checks look for.

```bash
npx skills add medusajs/medusa-agent-skills
```

[View the skill on GitHub](https://github.com/medusajs/medusa-agent-skills/tree/main/plugins/ecommerce-storefront/skills/storefront-best-practices)

---

Scan your store for free at https://is-agentic-storefront.com/