Which AI bots should an online store allow?

Allow AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot, Amzn-SearchBot, Meta-WebIndexer, DuckAssistBot) and user-initiated agents (ChatGPT-User, Claude-User, Perplexity-User, Google-Agent, Amzn-User, Meta-ExternalFetcher, MistralAI-User) on product and collection pages.

Does blocking GPTBot remove my store from ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search and shopping results use OAI-SearchBot, and live page lookups use ChatGPT-User. You can block GPTBot and still appear in ChatGPT as long as those two are allowed.

Do AI agents respect robots.txt?

Training and search crawlers generally do. Several user-initiated fetchers don't: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. For those agents, your WAF and bot-protection settings matter more than robots.txt.

How can I tell a real AI bot from a fake one?

Don't trust the user-agent string alone. Check the request IP against the vendor's published IP list or reverse DNS, or verify a Web Bot Auth signature. Cloudflare and other CDNs can do this for you as verified bots or signed agents.

What user agent do browser-based AI agents use?

Browser-based agents such as ChatGPT Atlas, Claude in Chrome, Perplexity Comet and Gemini in Chrome generally send a normal Chrome user agent, so robots.txt and user-agent rules don't apply to them. Only bot scoring and CAPTCHAs affect them.

AI crawler and agent user agents: the 2026 reference for stores

What kinds of AI bots visit an ecommerce store?

AI bots that visit an ecommerce store fall into three classes. The class decides whether blocking the bot costs you anything.

Training crawlers collect content to train AI models. Blocking them is a legitimate business choice and doesn’t remove you from AI answers.
AI search bots index pages so an assistant can cite or list them in answers and shopping results. Blocking them makes your products invisible in that assistant.
User-initiated agents fetch a page in real time because a person asked the assistant to. Blocking them breaks agentic shopping directly: the shopper asks about your product and the agent can’t load it.

The Agent-Friendly Ecommerce Store Scorer follows the same split: blocking training crawlers is info-only, while blocking search bots or user-initiated agents lowers the robots.txt and agent access scores.

Which AI user agents exist in 2026?

The table lists each AI user-agent token documented by its operator as of October 2026. “Respects robots.txt” reflects what the operator’s own documentation says. Vendors change this list several times a year (Google renamed its NotebookLM fetcher in 2026), so check the linked docs before relying on it.

Operator	User-agent token	Type	Respects robots.txt?	Official docs
OpenAI	`GPTBot`	Training	Yes	OpenAI bots
OpenAI	`OAI-SearchBot`	AI search (ChatGPT search and shopping)	Yes	OpenAI bots
OpenAI	`ChatGPT-User`	User-initiated	“robots.txt rules may not apply”	OpenAI bots
OpenAI	`OAI-AdsBot`	Ads landing-page check	Not stated	OpenAI bots
OpenAI	ChatGPT agent (cloud browser)	User-initiated, signs with Web Bot Auth	n/a (browser)	OpenAI allowlisting
Anthropic	`ClaudeBot`	Training	Yes, plus `Crawl-delay`	Anthropic crawlers
Anthropic	`Claude-SearchBot`	AI search	Yes	Anthropic crawlers
Anthropic	`Claude-User`	User-initiated	Yes	Anthropic crawlers
Perplexity	`PerplexityBot`	AI search (not training)	Yes	Perplexity bots
Perplexity	`Perplexity-User`	User-initiated	“Generally ignores”	Perplexity bots
Google	`Googlebot`	Search (also feeds AI Overviews and AI Mode)	Yes	Google crawlers
Google	`Google-Extended`	Training/grounding opt-out token (no separate crawler)	Yes	Google crawlers
Google	`Google-CloudVertexBot`	Crawls for Vertex AI Agents	Yes	Google crawlers
Google	`Google-Agent`	User-initiated agents on Google infrastructure	Generally ignores	Google user-triggered fetchers
Google	`Google-GeminiNotebook` (formerly `Google-NotebookLM`)	User-initiated	Generally ignores	Google user-triggered fetchers
Apple	`Applebot`	Search (Siri, Spotlight)	Yes; falls back to Googlebot rules	Apple
Apple	`Applebot-Extended`	Training opt-out token (no separate crawler)	Yes	Apple
Meta	`Meta-ExternalAgent`	Training	Yes	Meta crawlers
Meta	`Meta-WebIndexer`	AI search	Yes	Meta crawlers
Meta	`Meta-ExternalFetcher`	User-initiated	May bypass	Meta crawlers
Amazon	`Amazonbot`	Training and product use	Yes (no `Crawl-delay`)	Amazonbot
Amazon	`Amzn-SearchBot`	AI search (Alexa, Rufus), not training	Yes	Amazonbot
Amazon	`Amzn-User`	User-initiated	Yes	Amazonbot
Microsoft	`Bingbot`	Search (grounds Copilot)	Yes	Bing crawlers
Mistral	`MistralAI-User`	User-initiated	Not stated	Mistral robots
DuckDuckGo	`DuckAssistBot`	AI answers (not training)	Yes	DuckDuckGo

Google-Extended and Applebot-Extended are robots.txt tokens only. Disallowing them opts your content out of AI training without affecting Googlebot or Applebot crawling for search.

Why does the difference between training and search bots matter for stores?

The difference matters because each class does a different job for your store. A store that blocks every AI bot to stop training also disappears from ChatGPT search, Claude, Perplexity and Rufus answers, and shopping agents can’t open its product pages.

Block a training crawler (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent): your content isn’t used for model training. Answers and agent shopping are unaffected.
Block an AI search bot (OAI-SearchBot, Claude-SearchBot, PerplexityBot): your products drop out of that assistant’s answers and shopping results.
Block a user-initiated agent (ChatGPT-User, Claude-User, Perplexity-User): a shopper who asks “is this in stock in size 10?” gets no answer from your store.

Robots.txt is not the only place blocks happen. WAFs and CDNs often return 403, 429 or a challenge page to AI user agents regardless of robots.txt. Cloudflare has blocked AI crawlers by default on new zones since July 2025 (Cloudflare), and its pay-per-crawl feature answers with HTTP 402 (Cloudflare docs). The agent access check fetches your product page with each AI search and user-agent string to catch these.

What robots.txt should an online store use for AI bots?

A good store robots.txt allows AI search bots and user-initiated agents on product, collection and search pages, keeps cart and account paths private, and declares a sitemap. Opting out of training is optional. Rules follow RFC 9309: a bot that matches a named group ignores the * group, and the longest matching path wins.

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

# Optional: opt out of AI training. Not penalized by the scorer.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Notes for stores:

Disallow on /cart and /checkout is common (some hosted platforms do it by default) and isn’t penalized.
Don’t disallow /search. Agents use site search to find products.
A robots.txt that returns 5xx is treated as “disallow everything” under RFC 9309 §2.3.1. A 404 means “allow everything”.
Don’t add a blanket User-agent: * Disallow: / and then allow a few AI bots by name. Any agent you didn’t list is blocked.

The robots.txt check evaluates each search and user-agent token against your homepage and a sampled product URL.

How do I verify that an AI bot is real?

Verify AI bots by something other than the user-agent string, because anyone can send User-Agent: ChatGPT-User. There are two reliable methods.

IP lists and reverse DNS

Most operators publish the IP ranges their crawlers use, or a reverse DNS pattern, on the docs pages linked in the table above. Anthropic, for example, publishes its bot IPs at https://claude.com/crawling/bots.json (Anthropic). CDNs such as Cloudflare maintain these lists for you as “verified bots”, so an allow rule for verified AI search bots is usually one setting rather than a hand-maintained list.

Web Bot Auth (signed requests)

Web Bot Auth lets an agent sign each request with HTTP Message Signatures (RFC 9421), using Signature, Signature-Input and Signature-Agent headers. The agent publishes its public keys at /.well-known/http-message-signatures-directory on its own domain (IETF draft). The IETF webbotauth working group is standardizing it (IETF).

Who signs today:

ChatGPT agent signs with Signature-Agent: "https://chatgpt.com" (OpenAI, Cloudflare).
Google is experimenting with the identity https://agent.bot.goog for Google-Agent (Google).
AWS Bedrock AgentCore browser can sign requests (AWS).

Cloudflare treats signed agents as verified (Cloudflare docs), and Visa’s Trusted Agent Protocol and Mastercard Agent Pay build on Web Bot Auth (Cloudflare). Whether a store accepts signed agents is a server-side policy, so the scorer can’t see it and doesn’t score it directly. It scores the result: whether agent requests get through.

What about browser-based AI agents?

Browser-based AI agents such as ChatGPT Atlas, Claude in Chrome, Perplexity Comet and Gemini in Chrome generally present a normal Chrome user agent. Robots.txt and user-agent rules don’t apply to them. What stops them is bot scoring, challenge pages and CAPTCHAs: Google’s computer-use model must never solve a CAPTCHA (Gemini docs). The no-CAPTCHA check covers this, and agent-ready ecommerce explains how these agents read pages.

How does the scorer test AI bot access?

The scorer fetches your sampled product page once with a browser user agent and once with each AI search and user-initiated agent string, then compares status codes, challenge markers and whether the product content is still there. Because spoofed user agents from our IP aren’t the real bots, a generic bot challenge is reported as inconclusive, not fail. Details are in the scoring methodology.