# AI crawler and agent user agents: the 2026 reference for stores > AI bots fall into three classes: training crawlers (GPTBot, ClaudeBot, Google-Extended), AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-initiated agents (ChatGPT-User, Claude-User, Perplexity-User). Stores can block training crawlers without losing AI visibility, but blocking search bots or user agents hides products from AI answers and AI shopping agents. - Source: https://is-agentic-storefront.com/guides/ai-user-agents - Updated: 2026-10-15 ## What kinds of AI bots visit an ecommerce store? AI bots that visit an ecommerce store fall into three classes. The class decides whether blocking the bot costs you anything. - **Training crawlers** collect content to train AI models. Blocking them is a legitimate business choice and doesn't remove you from AI answers. - **AI search bots** index pages so an assistant can cite or list them in answers and shopping results. Blocking them makes your products invisible in that assistant. - **User-initiated agents** fetch a page in real time because a person asked the assistant to. Blocking them breaks agentic shopping directly: the shopper asks about your product and the agent can't load it. The Agent-Friendly Ecommerce Store Scorer follows the same split: blocking training crawlers is info-only, while blocking search bots or user-initiated agents lowers the [robots.txt](https://is-agentic-storefront.com/checks/robots-txt) and [agent access](https://is-agentic-storefront.com/checks/agent-access) scores. ## Which AI user agents exist in 2026? The table lists each AI user-agent token documented by its operator as of October 2026. "Respects robots.txt" reflects what the operator's own documentation says. Vendors change this list several times a year (Google renamed its NotebookLM fetcher in 2026), so check the linked docs before relying on it. | Operator | User-agent token | Type | Respects robots.txt? | Official docs | | ---------- | ------------------------------------------------------ | ------------------------------------------------------ | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | OpenAI | `GPTBot` | Training | Yes | [OpenAI bots](https://developers.openai.com/api/docs/bots) | | OpenAI | `OAI-SearchBot` | AI search (ChatGPT search and shopping) | Yes | [OpenAI bots](https://developers.openai.com/api/docs/bots) | | OpenAI | `ChatGPT-User` | User-initiated | "robots.txt rules may not apply" | [OpenAI bots](https://developers.openai.com/api/docs/bots) | | OpenAI | `OAI-AdsBot` | Ads landing-page check | Not stated | [OpenAI bots](https://developers.openai.com/api/docs/bots) | | OpenAI | ChatGPT agent (cloud browser) | User-initiated, signs with Web Bot Auth | n/a (browser) | [OpenAI allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-works-cloud-browser-allowlisting) | | Anthropic | `ClaudeBot` | Training | Yes, plus `Crawl-delay` | [Anthropic crawlers](https://support.claude.com/en/articles/8896518) | | Anthropic | `Claude-SearchBot` | AI search | Yes | [Anthropic crawlers](https://support.claude.com/en/articles/8896518) | | Anthropic | `Claude-User` | User-initiated | Yes | [Anthropic crawlers](https://support.claude.com/en/articles/8896518) | | Perplexity | `PerplexityBot` | AI search (not training) | Yes | [Perplexity bots](https://docs.perplexity.ai/guides/bots) | | Perplexity | `Perplexity-User` | User-initiated | "Generally ignores" | [Perplexity bots](https://docs.perplexity.ai/guides/bots) | | Google | `Googlebot` | Search (also feeds AI Overviews and AI Mode) | Yes | [Google crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) | | Google | `Google-Extended` | Training/grounding opt-out token (no separate crawler) | Yes | [Google crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) | | Google | `Google-CloudVertexBot` | Crawls for Vertex AI Agents | Yes | [Google crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) | | Google | `Google-Agent` | User-initiated agents on Google infrastructure | Generally ignores | [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) | | Google | `Google-GeminiNotebook` (formerly `Google-NotebookLM`) | User-initiated | Generally ignores | [Google user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers) | | Apple | `Applebot` | Search (Siri, Spotlight) | Yes; falls back to Googlebot rules | [Apple](https://support.apple.com/en-us/119829) | | Apple | `Applebot-Extended` | Training opt-out token (no separate crawler) | Yes | [Apple](https://support.apple.com/en-us/119829) | | Meta | `Meta-ExternalAgent` | Training | Yes | [Meta crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) | | Meta | `Meta-WebIndexer` | AI search | Yes | [Meta crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) | | Meta | `Meta-ExternalFetcher` | User-initiated | May bypass | [Meta crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/) | | Amazon | `Amazonbot` | Training and product use | Yes (no `Crawl-delay`) | [Amazonbot](https://developer.amazon.com/amazonbot) | | Amazon | `Amzn-SearchBot` | AI search (Alexa, Rufus), not training | Yes | [Amazonbot](https://developer.amazon.com/amazonbot) | | Amazon | `Amzn-User` | User-initiated | Yes | [Amazonbot](https://developer.amazon.com/amazonbot) | | Microsoft | `Bingbot` | Search (grounds Copilot) | Yes | [Bing crawlers](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) | | Mistral | `MistralAI-User` | User-initiated | Not stated | [Mistral robots](https://docs.mistral.ai/robots) | | DuckDuckGo | `DuckAssistBot` | AI answers (not training) | Yes | [DuckDuckGo](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) | `Google-Extended` and `Applebot-Extended` are robots.txt tokens only. Disallowing them opts your content out of AI training without affecting Googlebot or Applebot crawling for search. ## Why does the difference between training and search bots matter for stores? The difference matters because each class does a different job for your store. A store that blocks every AI bot to stop training also disappears from ChatGPT search, Claude, Perplexity and Rufus answers, and shopping agents can't open its product pages. - **Block a training crawler** (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent): your content isn't used for model training. Answers and agent shopping are unaffected. - **Block an AI search bot** (OAI-SearchBot, Claude-SearchBot, PerplexityBot): your products drop out of that assistant's answers and shopping results. - **Block a user-initiated agent** (ChatGPT-User, Claude-User, Perplexity-User): a shopper who asks "is this in stock in size 10?" gets no answer from your store. Robots.txt is not the only place blocks happen. WAFs and CDNs often return `403`, `429` or a challenge page to AI user agents regardless of robots.txt. Cloudflare has blocked AI crawlers by default on new zones since July 2025 ([Cloudflare](https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/)), and its pay-per-crawl feature answers with HTTP `402` ([Cloudflare docs](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/crawl-pages/)). The [agent access check](https://is-agentic-storefront.com/checks/agent-access) fetches your product page with each AI search and user-agent string to catch these. ## What robots.txt should an online store use for AI bots? A good store robots.txt allows AI search bots and user-initiated agents on product, collection and search pages, keeps cart and account paths private, and declares a sitemap. Opting out of training is optional. Rules follow [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html): a bot that matches a named group ignores the `*` group, and the longest matching path wins. ```txt User-agent: * Disallow: /cart Disallow: /checkout Disallow: /account # Optional: opt out of AI training. Not penalized by the scorer. User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Meta-ExternalAgent Disallow: / Sitemap: https://www.example.com/sitemap.xml ``` Notes for stores: - `Disallow` on `/cart` and `/checkout` is common (some hosted platforms do it by default) and isn't penalized. - Don't disallow `/search`. Agents use [site search](https://is-agentic-storefront.com/checks/site-search) to find products. - A robots.txt that returns `5xx` is treated as "disallow everything" under [RFC 9309 §2.3.1](https://www.rfc-editor.org/rfc/rfc9309.html#section-2.3.1). A `404` means "allow everything". - Don't add a blanket `User-agent: * Disallow: /` and then allow a few AI bots by name. Any agent you didn't list is blocked. The [robots.txt check](https://is-agentic-storefront.com/checks/robots-txt) evaluates each search and user-agent token against your homepage and a sampled product URL. ## How do I verify that an AI bot is real? Verify AI bots by something other than the user-agent string, because anyone can send `User-Agent: ChatGPT-User`. There are two reliable methods. ### IP lists and reverse DNS Most operators publish the IP ranges their crawlers use, or a reverse DNS pattern, on the docs pages linked in the table above. Anthropic, for example, publishes its bot IPs at `https://claude.com/crawling/bots.json` ([Anthropic](https://support.claude.com/en/articles/8896518)). CDNs such as Cloudflare maintain these lists for you as "verified bots", so an allow rule for verified AI search bots is usually one setting rather than a hand-maintained list. ### Web Bot Auth (signed requests) Web Bot Auth lets an agent sign each request with [HTTP Message Signatures (RFC 9421)](https://www.rfc-editor.org/rfc/rfc9421.html), using `Signature`, `Signature-Input` and `Signature-Agent` headers. The agent publishes its public keys at `/.well-known/http-message-signatures-directory` on its own domain ([IETF draft](https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/)). The IETF `webbotauth` working group is standardizing it ([IETF](https://datatracker.ietf.org/wg/webbotauth/about/)). Who signs today: - ChatGPT agent signs with `Signature-Agent: "https://chatgpt.com"` ([OpenAI](https://help.openai.com/en/articles/11845367-chatgpt-works-cloud-browser-allowlisting), [Cloudflare](https://blog.cloudflare.com/signed-agents/)). - Google is experimenting with the identity `https://agent.bot.goog` for `Google-Agent` ([Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers)). - AWS Bedrock AgentCore browser can sign requests ([AWS](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/browser-web-bot-auth.html)). Cloudflare treats signed agents as verified ([Cloudflare docs](https://developers.cloudflare.com/bots/concepts/bot/signed-agents/)), and Visa's Trusted Agent Protocol and Mastercard Agent Pay build on Web Bot Auth ([Cloudflare](https://blog.cloudflare.com/secure-agentic-commerce/)). Whether a store accepts signed agents is a server-side policy, so the scorer can't see it and doesn't score it directly. It scores the result: whether agent requests get through. ## What about browser-based AI agents? Browser-based AI agents such as ChatGPT Atlas, Claude in Chrome, Perplexity Comet and Gemini in Chrome generally present a normal Chrome user agent. Robots.txt and user-agent rules don't apply to them. What stops them is bot scoring, challenge pages and CAPTCHAs: Google's computer-use model must never solve a CAPTCHA ([Gemini docs](https://ai.google.dev/gemini-api/docs/computer-use)). The [no-CAPTCHA check](https://is-agentic-storefront.com/checks/no-captcha) covers this, and [agent-ready ecommerce](https://is-agentic-storefront.com/guides/agent-ready-ecommerce) explains how these agents read pages. ## How does the scorer test AI bot access? The scorer fetches your sampled product page once with a browser user agent and once with each AI search and user-initiated agent string, then compares status codes, challenge markers and whether the product content is still there. Because spoofed user agents from our IP aren't the real bots, a generic bot challenge is reported as `inconclusive`, not `fail`. Details are in the [scoring methodology](https://is-agentic-storefront.com/guides/scoring-methodology). Related: [robots.txt glossary entry](https://is-agentic-storefront.com/glossary#robots-txt) · [agent readiness vs SEO](https://is-agentic-storefront.com/guides/agent-readiness-vs-seo) ## Frequently asked questions ### Which AI bots should an online store allow? Allow AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot, Amzn-SearchBot, Meta-WebIndexer, DuckAssistBot) and user-initiated agents (ChatGPT-User, Claude-User, Perplexity-User, Google-Agent, Amzn-User, Meta-ExternalFetcher, MistralAI-User) on product and collection pages. ### Does blocking GPTBot remove my store from ChatGPT? No. GPTBot is OpenAI's training crawler. ChatGPT search and shopping results use OAI-SearchBot, and live page lookups use ChatGPT-User. You can block GPTBot and still appear in ChatGPT as long as those two are allowed. ### Do AI agents respect robots.txt? Training and search crawlers generally do. Several user-initiated fetchers don't: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. For those agents, your WAF and bot-protection settings matter more than robots.txt. ### How can I tell a real AI bot from a fake one? Don't trust the user-agent string alone. Check the request IP against the vendor's published IP list or reverse DNS, or verify a Web Bot Auth signature. Cloudflare and other CDNs can do this for you as verified bots or signed agents. ### What user agent do browser-based AI agents use? Browser-based agents such as ChatGPT Atlas, Claude in Chrome, Perplexity Comet and Gemini in Chrome generally send a normal Chrome user agent, so robots.txt and user-agent rules don't apply to them. Only bot scoring and CAPTCHAs affect them. ## Run your store on a platform built for agentic commerce Medusa is open-source, headless commerce with a full Store API agents can call and storefronts you control end to end. ### Deploy on Medusa Cloud Open-source commerce with a full Store API, hosted and scaled for you. Your storefront, your data model. - [Deploy on Medusa Cloud](https://cloud.medusajs.com?utm_source=agent-friendly-scorer&utm_medium=referral&utm_campaign=markdown-cta&utm_content=%2Fguides%2Fai-user-agents) - [Medusa Cloud docs](https://docs.medusajs.com/cloud?utm_source=agent-friendly-scorer&utm_medium=referral&utm_campaign=markdown-cta&utm_content=%2Fguides%2Fai-user-agents) ### Storefront skills for coding agents Teach Claude Code, Cursor or Codex to build storefronts with the patterns these checks look for. ```bash npx skills add medusajs/medusa-agent-skills ``` [View the skill on GitHub](https://github.com/medusajs/medusa-agent-skills/tree/main/plugins/ecommerce-storefront/skills/storefront-best-practices) --- Scan your store for free at https://is-agentic-storefront.com/