robots.txt for AI agents and AI search bots

Why robots.txt matters for AI agents

robots.txt is the first file AI crawlers read. Each vendor publishes the user-agent tokens it honors, and blocking the wrong one hides your catalog from AI answers:

• AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot) build the indexes behind ChatGPT search, Claude, Perplexity, Google AI Overviews and Copilot. Blocked here means absent from those answers.
• User-initiated agents (ChatGPT-User, Claude-User, Perplexity-User, Google-Agent) fetch a page because a person asked. Blocking them breaks a live shopping task.
• Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot) collect model training data. Opting out is a legitimate business choice.

A common accident is a blanket “block all AI” snippet copied from a blog, which disallows search bots and user agents along with training crawlers. The tokens and their purposes are listed in AI user agents explained.

What the scanner checks

The scanner fetches /robots.txt and parses it per RFC 9309: case-insensitive token match, groups for the same token are merged, the * group applies only when no group names the bot, the longest matching path wins, and Allow wins ties. Wildcards (*) and end anchors ($) are supported.

It then evaluates 22 AI and search tokens against two paths: / and the path of a sampled product page.

Result	Condition
Pass	All 16 search and user-initiated tokens may fetch / and the product path. Also passes when robots.txt is missing (4xx, or an HTML page served instead), since that means “allow all”.
Partial	Fewer than 8 of the 16 search/user tokens are disallowed. Scored proportionally.
Fail	8 or more search/user tokens are disallowed, or robots.txt returns a 5xx or is unreachable (RFC 9309 treats that as “disallow all”).

Blocked training crawlers are listed as information, not penalized. A Disallow that covers /search for ChatGPT-User is also reported, because agents use site search (see site search).

How to fix robots.txt for AI agents

Give search bots and user-initiated agents an explicit group, keep the cart and checkout rules you need, and block training separately if you want to:

# AI search and user-initiated agents: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /cart
Disallow: /checkout
Disallow: /account

# AI training: opted out (optional, not penalized)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /

# Everyone else
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

Sitemap: https://your-store.com/sitemap.xml

Things to watch:

1. A named group replaces * for that bot. Under RFC 9309, once OAI-SearchBot has its own group, it ignores the * group entirely. Copy any rules you still need into the named group.
2. Serve it as text/plain with HTTP 200. A 5xx response, or a timeout, is read as “disallow everything”.
3. Add a Sitemap: line with an absolute URL so crawlers find product URLs (see sitemap).
4. Don’t rely on robots.txt alone. OpenAI and Perplexity say their user-initiated agents may not apply robots.txt rules, and a WAF can still block a bot robots.txt allows. Check agent access too.

Verify with any RFC 9309 tester, or run curl -i https://your-store.com/robots.txt and confirm 200 and content-type: text/plain.

How to fix it on Medusa

• Not shipped by default. The Medusa DTC starter storefront has no robots.txt, so the check passes as “allow all”, but crawlers get no Sitemap: line.
• Add app/robots.ts to the Next.js storefront with the Next.js Metadata API, or a static public/robots.txt. Point its sitemap at the storefront’s app/sitemap.ts (see sitemap).
• See Is Medusa agent-ready? for the full starter checklist.

Common mistakes

• Copying a “block all AI bots” list that includes OAI-SearchBot, PerplexityBot or ChatGPT-User.
• User-agent: * / Disallow: / left over from a staging environment.
• Disallowing /products/ or category paths to save crawl budget.
• Returning an HTML 200 page at /robots.txt from a catch-all route. The scanner treats it as missing, but some crawlers may not.

Sources

• RFC 9309: Robots Exclusion Protocol
• OpenAI crawlers and user agents
• Anthropic crawlers
• Perplexity crawlers
• Google common crawlers
• Google user-triggered fetchers
• Applebot
• Next.js: robots.txt file convention