# robots.txt for AI agents and AI search bots

> An agent-ready robots.txt lets AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-initiated agents (ChatGPT-User, Claude-User, Perplexity-User) read your homepage and product pages. You can still block AI training crawlers like GPTBot and ClaudeBot without losing visibility in AI shopping answers.

- Source: https://is-agentic-storefront.com/checks/robots-txt
- Updated: 2026-10-15
- Category: Access (25% of score)
- Impact: High
- Spec: [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)
- Scanner check: robots.txt allows AI agents and AI search bots

## Why robots.txt matters for AI agents

robots.txt is the first file AI crawlers read. Each vendor publishes the user-agent tokens it honors, and blocking the wrong one hides your catalog from AI answers:

- **AI search bots** (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Googlebot`, `Bingbot`, `Applebot`) build the indexes behind ChatGPT search, Claude, Perplexity, Google AI Overviews and Copilot. Blocked here means absent from those answers.
- **User-initiated agents** (`ChatGPT-User`, `Claude-User`, `Perplexity-User`, `Google-Agent`) fetch a page because a person asked. Blocking them breaks a live shopping task.
- **Training crawlers** (`GPTBot`, `ClaudeBot`, `Google-Extended`, `Applebot-Extended`, `Meta-ExternalAgent`, `Amazonbot`) collect model training data. Opting out is a legitimate business choice.

A common accident is a blanket "block all AI" snippet copied from a blog, which disallows search bots and user agents along with training crawlers. The tokens and their purposes are listed in [AI user agents explained](https://is-agentic-storefront.com/guides/ai-user-agents).

## What the scanner checks

The scanner fetches `/robots.txt` and parses it per [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html): case-insensitive token match, groups for the same token are merged, the `*` group applies only when no group names the bot, the longest matching path wins, and `Allow` wins ties. Wildcards (`*`) and end anchors (`$`) are supported.

It then evaluates 22 AI and search tokens against two paths: `/` and the path of a sampled product page.

| Result  | Condition                                                                                                                                                                               |
| ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Pass    | All 16 search and user-initiated tokens may fetch `/` and the product path. Also passes when robots.txt is missing (4xx, or an HTML page served instead), since that means "allow all". |
| Partial | Fewer than 8 of the 16 search/user tokens are disallowed. Scored proportionally.                                                                                                        |
| Fail    | 8 or more search/user tokens are disallowed, or robots.txt returns a 5xx or is unreachable (RFC 9309 treats that as "disallow all").                                                    |

Blocked training crawlers are listed as information, not penalized. A `Disallow` that covers `/search` for `ChatGPT-User` is also reported, because agents use site search (see [site search](https://is-agentic-storefront.com/checks/site-search)).

## How to fix robots.txt for AI agents

Give search bots and user-initiated agents an explicit group, keep the cart and checkout rules you need, and block training separately if you want to:

```text
# AI search and user-initiated agents: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /cart
Disallow: /checkout
Disallow: /account

# AI training: opted out (optional, not penalized)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /

# Everyone else
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

Sitemap: https://your-store.com/sitemap.xml
```

Things to watch:

1. **A named group replaces `*` for that bot.** Under RFC 9309, once `OAI-SearchBot` has its own group, it ignores the `*` group entirely. Copy any rules you still need into the named group.
2. **Serve it as `text/plain` with HTTP 200.** A 5xx response, or a timeout, is read as "disallow everything".
3. **Add a `Sitemap:` line** with an absolute URL so crawlers find product URLs (see [sitemap](https://is-agentic-storefront.com/checks/sitemap)).
4. **Don't rely on robots.txt alone.** OpenAI and Perplexity say their user-initiated agents may not apply robots.txt rules, and a WAF can still block a bot robots.txt allows. Check [agent access](https://is-agentic-storefront.com/checks/agent-access) too.

Verify with any RFC 9309 tester, or run `curl -i https://your-store.com/robots.txt` and confirm `200` and `content-type: text/plain`.

## How to fix it on Medusa

- **Not shipped by default.** The Medusa [DTC starter](https://github.com/medusajs/dtc-starter) storefront has no robots.txt, so the check passes as "allow all", but crawlers get no `Sitemap:` line.
- **Add `app/robots.ts`** to the Next.js storefront with the [Next.js Metadata API](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots), or a static `public/robots.txt`. Point its `sitemap` at the storefront's `app/sitemap.ts` (see [sitemap](https://is-agentic-storefront.com/checks/sitemap)).
- See [Is Medusa agent-ready?](https://is-agentic-storefront.com/guides/medusa-agent-readiness) for the full starter checklist.

## Common mistakes

- Copying a "block all AI bots" list that includes `OAI-SearchBot`, `PerplexityBot` or `ChatGPT-User`.
- `User-agent: *` / `Disallow: /` left over from a staging environment.
- Disallowing `/products/` or category paths to save crawl budget.
- Returning an HTML 200 page at `/robots.txt` from a catch-all route. The scanner treats it as missing, but some crawlers may not.

## Sources

- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)
- [OpenAI crawlers and user agents](https://developers.openai.com/api/docs/bots)
- [Anthropic crawlers](https://support.claude.com/en/articles/8896518)
- [Perplexity crawlers](https://docs.perplexity.ai/guides/bots)
- [Google common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
- [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers)
- [Applebot](https://support.apple.com/en-us/119829)
- [Next.js: robots.txt file convention](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots)

## Frequently asked questions

### Can I block GPTBot and still appear in ChatGPT shopping results?

Yes. GPTBot collects training data, while OAI-SearchBot powers ChatGPT search and ChatGPT-User fetches pages for a user. Block GPTBot and allow the other two.

### What happens if my robots.txt returns a 500 error?

Under RFC 9309, crawlers treat a server error on robots.txt as "disallow everything". The scanner fails the check for the same reason.

### Is it bad to have no robots.txt at all?

No. A 404 means "allow everything" under RFC 9309, so the check passes. Adding one with a Sitemap line still helps crawlers find your product URLs.

### Should I disallow /cart and /checkout?

That is normal and not penalized. Disallowing /search is flagged, because agents use site search to find products.

## Run your store on a platform built for agentic commerce

Medusa is open-source, headless commerce with a full Store API agents can call and storefronts you control end to end.

### Deploy on Medusa Cloud

Open-source commerce with a full Store API, hosted and scaled for you. Your storefront, your data model.

- [Deploy on Medusa Cloud](https://cloud.medusajs.com?utm_source=agent-friendly-scorer&utm_medium=referral&utm_campaign=markdown-cta&utm_content=%2Fchecks%2Frobots-txt)
- [Medusa Cloud docs](https://docs.medusajs.com/cloud?utm_source=agent-friendly-scorer&utm_medium=referral&utm_campaign=markdown-cta&utm_content=%2Fchecks%2Frobots-txt)

### Storefront skills for coding agents

Teach Claude Code, Cursor or Codex to build storefronts with the patterns these checks look for.

```bash
npx skills add medusajs/medusa-agent-skills
```

[View the skill on GitHub](https://github.com/medusajs/medusa-agent-skills/tree/main/plugins/ecommerce-storefront/skills/storefront-best-practices)

---

Scan your store for free at https://is-agentic-storefront.com/