Canonical URLs and soft 404s on product pages for AI agents

Why it matters for AI agents

AI agents act on what URLs claim to be. Two signals tell them whether a product URL is real and which version is authoritative: the HTTP status code and the canonical link.

A canonical URL (<link rel="canonical">) names the preferred URL for a page when the same product is reachable through several paths, such as collection-scoped URLs, tracking parameters or variant query strings (Google canonical guidance). Agents and AI search systems use it to deduplicate products and to cite a stable link.

A soft 404 is a missing page that answers HTTP 200. Google treats this as an error (soft 404 errors) because clients cannot tell a removed product from a live one. For an AI shopping agent, that means recommending a product that no longer exists, or reading the homepage as if it were the product a shopper asked about.

What the scanner checks

The scanner tests up to three sampled product pages (from your sitemap or homepage links), then probes 404 handling.

• Canonical: read from <link rel="canonical"> in the HTML, or from a Link: <…>; rel="canonical" response header. Flagged when it is missing, points to the homepage (/), or points to a different domain (www. is ignored).
• Soft 404 probe: the scanner takes the first sampled product URL, replaces its last path segment with a random handle like agent-check-1a2b3c4d, and fetches it. Redirects are followed. A final 404 or 410 is correct; 200 is a soft 404; any other status is flagged.

Result	When
Pass	Every sampled product page has an acceptable canonical, and the fake product URL returns 404 or 410
Partial	Some issues: missing or wrong canonicals on some pages, a non-404 error status, or a soft 404 while canonicals are fine
Fail	The fake product URL returns 200 and every sampled product page has a canonical problem
Inconclusive	No product page could be loaded (no product URLs found, or product pages were blocked by a bot challenge)

How to fix it

1. Add a canonical link to every product page, using its own absolute URL (or the base product URL for variant URLs):

<link rel="canonical" href="https://shop.example.com/products/merino-crew-sock" />

2. Return HTTP 404 for unknown product handles, or 410 for products removed permanently. Do not render a “not found” message with 200, and do not redirect missing products to the homepage or a collection.
3. Keep out-of-stock products at HTTP 200 with "availability": "https://schema.org/OutOfStock" in the Product JSON-LD, so agents know the item exists but can’t be bought right now (see product structured data).
4. Verify with curl:

curl -o /dev/null -s -w "%{http_code}\n" https://shop.example.com/products/does-not-exist
# 404

Common mistakes

• Every page canonicalizes to the homepage. This usually comes from a theme or SEO plugin with a hardcoded site URL. It tells clients that all products are duplicates of the homepage.
• Canonical points to a staging or old domain after a migration.
• Catch-all routes in single-page apps that serve the app shell with 200 for any path, then render “not found” in JavaScript. The status code is decided on the server, so client-side “not found” views are always soft 404s.
• Redirecting missing products to search. It feels helpful, but the final response is a 200 page, so it is still a soft 404.

How to fix it on Medusa

In Medusa’s DTC starter, the product route already calls notFound() for unknown handles, so Next.js returns a real 404 (Next.js notFound). Keep that behavior if you customize the route, and in a custom storefront call notFound() whenever the Store API returns no product for the handle.

The starter sets a canonical on category pages but not on product pages. Add one with the alternates.canonical field in the product page’s generateMetadata (Next.js metadata). See the Medusa agent-readiness guide for the full checklist.

Sources

• Google: consolidate duplicate URLs (canonicals)
• Google: HTTP status codes and network errors
• Google: soft 404 errors
• RFC 8288: Web Linking (Link header)
• Next.js notFound()