How do AI agents crawl and read my website content?

Crawlers versus agents

A crawler fetches your HTML on a schedule and stores it for later — that is how a model learns your product exists. An agent arrives with a goal right now: find a price, book a slot, add something to a basket. It reads the page, decides on an action, takes it, and reads what happened next.

A site can be perfectly crawlable and still be unusable by an agent trying to complete a task.

What makes content invisible

Most agent-facing problems come down to information that exists visually but not structurally.

• Content rendered only after client-side JavaScript, when the fetcher does not execute it.
• Prices, availability or specs shown as images rather than text.
• Key detail hidden behind a hover, a carousel, or a tab that requires interaction to reveal.
• Infinite scroll with no paginated URLs behind it.
• Critical context in a PDF with no HTML equivalent.

What actually helps

Nothing exotic. The same discipline that makes a site accessible makes it agent-readable.

• Server-render the content that matters, so it is in the initial HTML.
• Use real semantic elements — headings, lists, tables, labelled form fields.
• Publish structured data for products, prices, availability, hours and locations.
• Give every meaningful state its own URL.
• Keep robots.txt deliberate: know which agents you are allowing and which you are not.

Verifying it

Assumptions are cheap and usually wrong. PlateTale runs agents through your real pages and reports exactly what they could and could not read, and where a task fell apart.