Store speed for AI agents: response time and HTML page weight

Why it matters for AI agents

AI agents read stores the way a fast, impatient client does: they fetch HTML, extract what they need, and move to the next page under a time budget. Browser agents add a rendering step on top. Every extra second of server time and every extra kilobyte of HTML is paid for in latency and tokens.

Two numbers matter most:

• Time to first byte (TTFB), the time until the server starts responding. web.dev treats 0.8 seconds or less as good. A slow TTFB usually means pages are rendered per request with uncached backend calls.
• HTML weight. A product page that ships 1 MB of HTML, mostly inlined JSON state and scripts, forces an agent to wade through markup before it reaches the name, price and add-to-cart button. Some agents and tools truncate long pages, so product facts near the end can be lost.

No AI vendor publishes fetch timeouts or size limits, so these thresholds are heuristics, documented in the scoring methodology.

What the scanner checks

The scanner measures the homepage and up to three sampled product pages that loaded successfully. For each, it records the time until response headers arrive, then takes the median. It also finds the largest HTML document. Bodies are read up to a 2 MB cap.

Result	Median response time	Notes
Pass	under 800 ms	and the largest HTML is 1 MB or less
Partial	800 ms to under 2 s	or under 800 ms but HTML over 1 MB
Fail	2 s or more	or 800 ms to 2 s with HTML over 1 MB
Inconclusive	—	no page loaded to measure

HTML over 1 MB (or hitting the 2 MB read cap) drops the result one level. Measurements are taken from the scanner’s US region, so stores served far from the US without a CDN will look slower, which is also what a US-hosted agent would see.

How to fix it

1. Cache HTML at the CDN. Product and collection pages change rarely compared to how often they are read. Serve them from the edge with stale-while-revalidate so most requests never reach your origin:

Cache-Control: public, s-maxage=300, stale-while-revalidate=86400

2. Remove per-request backend waterfalls. Fetch product, pricing and inventory data in parallel, and cache Store API or database responses.
3. Trim the HTML. Remove duplicated inlined JSON (full product catalogs in hydration state, unused translations), move large inline scripts and styles to cached external files, and lazy-load below-the-fold sections like reviews and recommendations.
4. Enable compression. Serve HTML with gzip or Brotli; the scanner requests Accept-Encoding: gzip, br.
5. Verify with curl:

curl -s -o /dev/null -w "TTFB %{time_starttransfer}s, %{size_download} bytes\n" \
  https://shop.example.com/products/merino-crew-sock

Aim for under 0.8 s and under 1 MB. Run it a few times; the first request after a cache purge is always slower.

Common mistakes

• Server-rendering everything twice: the full HTML plus the same data serialized into a hydration blob. Keep the serialized state to what the client actually needs.
• Personalized HTML that can’t be cached, such as cart counts or geo pricing rendered on the server for every visitor. Load those parts client-side and keep the product HTML cacheable.
• Testing only from your own region. A store hosted in one region with no CDN can be fast locally and slow for agents running elsewhere.

Fast pages do not help if the content isn’t in the HTML at all; pair this with product facts in server HTML.

Sources

• web.dev: Optimize Time to First Byte
• web.dev: Core Web Vitals
• MDN: Cache-Control
• RFC 5861: stale-while-revalidate