# XML sitemap for AI agents: list every live product URL > An agent-ready ecommerce sitemap is an XML file, declared with a Sitemap line in robots.txt, that lists every live product URL with a lastmod date. AI agents and crawlers use it to enumerate your catalog instead of guessing links, so dead or missing product URLs mean products that agents never see. - Source: https://is-agentic-storefront.com/checks/sitemap - Updated: 2026-10-15 - Category: Discovery (15% of score) - Impact: High - Spec: [sitemaps.org protocol](https://www.sitemaps.org/protocol.html) - Scanner check: Sitemap is declared in robots.txt and lists live product URLs ## Why it matters for AI agents An XML sitemap is the catalog index for machines. When an AI shopping agent or an AI search crawler needs to know which products a store sells, the [sitemaps.org protocol](https://www.sitemaps.org/protocol.html) gives it a single file of canonical URLs with modification dates, instead of forcing it to follow navigation menus and JavaScript-rendered collection grids. For ecommerce, the product URLs are what matter. A sitemap that lists only blog posts and CMS pages, or one full of product URLs that redirect or 404, tells an agent your catalog is smaller or more broken than it is. The scanner also depends on the sitemap: it samples product pages from it, so a good sitemap makes every product-level check in the [scoring methodology](https://is-agentic-storefront.com/guides/scoring-methodology) more accurate. ## What the scanner checks The scanner reads `Sitemap:` lines in `/robots.txt`. If there are none, it tries `/sitemap.xml` and `/sitemap_index.xml`. Sitemaps with "product" or "pdp" in the URL are tried first (up to three root sitemaps, and up to two child sitemaps of a sitemap index). Gzipped sitemaps are skipped, and it reads at most 5,000 URLs. A URL counts as a product URL when it comes from a product-named sitemap or its path looks like a product path (`/products/…`, `/product/…`, `/p/…`, `/dp/…`, `/item/…`). The scanner then fetches up to three sampled product URLs. | Result | When | | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Pass** | Sitemap declared in robots.txt, product URLs present, entries have ``, all product URLs on your host, sampled product URLs return 200 without redirects | | **Partial** | A sitemap with product URLs exists, but one or more of: not declared in robots.txt, no ``, product URLs on another host, some sampled URLs fail or redirect | | **Fail** | No sitemap found, no product URLs in it, or every sampled product URL fails to return 200 | This is a high-weight check in the Discovery category. ## How to fix it 1. Generate an XML sitemap that includes every published product and collection URL, each with ``. Use absolute, canonical URLs on your own host that return 200 directly, with no redirects. 2. Split large catalogs with a sitemap index. Each file can hold at most 50,000 URLs and 50 MB uncompressed. Put "product" in the product sitemap's file name so clients can find it quickly. 3. Declare it in robots.txt: ```txt User-agent: * Allow: / Sitemap: https://shop.example.com/sitemap.xml ``` 4. Regenerate the sitemap whenever products are added, removed or renamed. A minimal product sitemap: ```xml https://shop.example.com/products/merino-crew-sock 2026-10-01 ``` Then check a few entries with `curl -I ` and confirm each prints `HTTP/2 200`, not 301 or 404. Product URLs that redirect usually mean the sitemap lists old handles or non-canonical variants; see [canonical URLs and status codes](https://is-agentic-storefront.com/checks/canonical-status). ## Common mistakes - **Listing the help-center or blog sitemap first** in robots.txt with no product sitemap at all. The scanner tries product-named sitemaps first, but it can only find products you list. - **Stale handles** that 301 to the renamed product. Agents waste requests, and the scanner marks the sitemap partial. - **Product URLs on a different host**, such as a staging domain or old CDN hostname. - **Only a gzipped sitemap** (`sitemap.xml.gz`). It is valid per the protocol, but the scanner skips it; serve an uncompressed index too. ## How to fix it on Medusa Medusa's [DTC starter](https://github.com/medusajs/dtc-starter) storefront does not ship a sitemap or robots.txt, so add both in the Next.js app: - Add `app/sitemap.ts` that pages through the Store API's [list products route](https://docs.medusajs.com/resources/storefront-development/products/list) and returns one `/{countryCode}/products/{handle}` URL per product, each with a `lastModified` date ([Next.js sitemap](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap)). - Store API requests need the `x-publishable-api-key` header ([publishable API keys](https://docs.medusajs.com/resources/storefront-development/publishable-api-keys)). - Add `app/robots.ts` with a `sitemap` field pointing at the sitemap URL ([Next.js robots](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots)). The [Medusa agent-readiness guide](https://is-agentic-storefront.com/guides/medusa-agent-readiness) covers the other files the starter is missing. ## Sources - [Sitemaps XML protocol](https://www.sitemaps.org/protocol.html) - [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html) - [Next.js sitemap file convention](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/sitemap) - [Next.js robots file convention](https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots) - [Medusa: list products](https://docs.medusajs.com/resources/storefront-development/products/list) ## Frequently asked questions ### Do AI agents actually read sitemap.xml? Search and AI crawlers use sitemaps to enumerate URLs, and the sitemap is the most reliable way for any automated client to find product pages without crawling every link. Our scanner also uses it to pick which product pages to test. ### Does my sitemap need to be listed in robots.txt? The sitemaps.org protocol lets you declare it with a Sitemap line in robots.txt, and that is where clients look first. A sitemap only at /sitemap.xml still works, but the scanner scores it partial. ### Should out-of-stock products stay in the sitemap? Yes, as long as the product page still returns HTTP 200 and marks availability as OutOfStock. Remove URLs only when the product page is gone and returns 404 or 410. ## Run your store on a platform built for agentic commerce Medusa is open-source, headless commerce with a full Store API agents can call and storefronts you control end to end. ### Deploy on Medusa Cloud Open-source commerce with a full Store API, hosted and scaled for you. Your storefront, your data model. - [Deploy on Medusa Cloud](https://cloud.medusajs.com?utm_source=agent-friendly-scorer&utm_medium=referral&utm_campaign=markdown-cta&utm_content=%2Fchecks%2Fsitemap) - [Medusa Cloud docs](https://docs.medusajs.com/cloud?utm_source=agent-friendly-scorer&utm_medium=referral&utm_campaign=markdown-cta&utm_content=%2Fchecks%2Fsitemap) ### Storefront skills for coding agents Teach Claude Code, Cursor or Codex to build storefronts with the patterns these checks look for. ```bash npx skills add medusajs/medusa-agent-skills ``` [View the skill on GitHub](https://github.com/medusajs/medusa-agent-skills/tree/main/plugins/ecommerce-storefront/skills/storefront-best-practices) --- Scan your store for free at https://is-agentic-storefront.com/