Designing a Website for AI Agents
llms.txt, structured data, markdown renditions and MCP: what an AI-readable site actually requires.
By Mireille Kaplan, Contributing Analyst, Data Infrastructure
Two audiences, two representations
A modern site serves human readers who want typography and layout, and machine readers who want structure and want it cheaply. Serving the second group HTML built for the first is wasteful on both sides: the model spends tokens parsing navigation chrome, and the site loses control over what gets extracted.
The practical answer is to publish a parallel representation. Every substantive page should have a clean markdown rendition at a predictable URL, declared from the HTML version so it can be discovered rather than guessed at.
llms.txt is a table of contents, not a robots file
robots.txt is restrictive: it says what crawlers may not touch. llms.txt is the opposite: a curated markdown index telling a model what is worth reading and how the site is organised.
A useful llms.txt is short, links to the highest-value sections, and describes what each contains in a sentence. A companion llms-full.txt goes further, inlining substantive content so a model can ingest the site's core material in one fetch.
Structured data is now an extraction contract
Schema.org markup was originally about rich results in search. It now functions as an extraction contract for answer engines, which prefer explicitly typed data over inference from prose.
The types that matter most depend on your content: Organization and Product for entity pages, DefinedTerm for glossary entries, FAQPage for question content, BreadcrumbList everywhere, and ItemList for rankings. Mark up what is on the page and nothing more.
Make the site addressable by agents
Beyond documents, agents need endpoints. A JSON catalogue describing available data, a snapshot endpoint with current statistics, and an MCP server card describing callable tools turn a site from a document collection into something an agent can query.
The cost of publishing these is low and the discovery benefit compounds: an agent that can enumerate your data will cite it more accurately than one reconstructing it from rendered pages.
Do not neglect the human path
None of this replaces conventional SEO. Canonical discipline, internal linking, fast rendering and honest headings still determine whether pages get crawled and ranked at all, and AI systems overwhelmingly discover pages through the same indexes.
AI readability is an additional layer on a well-built site, not a substitute for one.
Frequently asked questions
- What is llms.txt?
- A markdown file at a site's root that gives language models a curated index of the site's most useful content and links to clean text versions of key pages. It is advisory, not enforced.
- Does publishing markdown versions of pages cause duplicate content problems?
- Not if the markdown renditions are excluded from indexing or canonicalised to their HTML equivalents. They exist for machine consumption, not for search ranking.
- Is MCP relevant to a content website?
- It becomes relevant when your data is queryable. Publishing an MCP server card lets agents call structured lookups instead of scraping pages, which produces more accurate citations.