What llms.txt actually is
llms.txt is a proposed open standard for a plain-text markdown file at the root of a domain (yoursite.com/llms.txt) that gives AI systems a curated guide to the most important content on the site. Think of it as a hand-built index, written for AI consumption.
The file format is intentionally simple: markdown, a top-level H1 for the site name, a short site description, then section headings for content categories with a list of canonical URLs under each. AI systems that support the standard fetch the file, read the index, and use it to decide which pages to read in full when answering a user question.
Three things llms.txt is not. First, it is not a crawler-permissions file. It does not say "allow this bot" or "block that bot"; that is what robots.txt does. Second, it is not a complete URL list like an XML sitemap. It is a curated subset of the most important pages. Third, it is not a security boundary. It only references content that is already public on the site.
The standard was proposed in late 2024 by Answer.AI and has been picked up by a range of smaller AI tools and documentation platforms. It sits in the same lineage as robots.txt and humans.txt: a simple, opt-in convention for sharing metadata about the site with consumers that want to read it.
The adoption picture in 2026
Honest summary of where adoption sits as of mid-2026.
Supported today
- Several AI documentation assistants and developer tools (Mintlify, Cursor and similar code-focused AI clients).
- A growing number of niche AI search products and internal AI tooling that index llms.txt as part of their crawl.
- Independent AI-powered research tools and browser plugins.
Not formally committed
- OpenAI (ChatGPT). No formal support; behaviour appears inconsistent in our testing.
- Google (Gemini, AI Overviews). No formal support.
- Anthropic (Claude). No formal support.
- Perplexity. No formal support.
The trajectory
Adoption has been increasing through 2025 and 2026, particularly among AI tools that work with technical and documentation-heavy sites. Whether the major vendors formally adopt the standard is still open. Our working position: the file is so cheap to add that waiting for formal adoption is the wrong call. Better to have it in place when adoption broadens than to be the site without it.
The cheap-insurance argument
Why we add llms.txt to every client site even though the major vendors have not formally adopted it.
- Trivial cost. The file takes 15 minutes to write for a small business. Hosting it is free. No ongoing maintenance beyond updating it when major pages change.
- No downside. Adding llms.txt does not affect your traditional SEO. The file is invisible to users and ignored by AI systems that do not support it. The worst case is no benefit.
- Real upside on partial adoption. Even today, the niche AI tools that read llms.txt include some that surface content to high-intent users. Being available in those tools costs nothing and provides modest incremental visibility.
- Position-of-readiness on broader adoption. If OpenAI, Google or Anthropic formally adopt the standard later this year or next, sites that already have llms.txt benefit immediately. Sites that have to add it then are catching up.
- Signal of seriousness. A site with a thoughtfully-written llms.txt sends a small signal that the operators understand how AI search works. That is worth something to humans inspecting the site, even if AI systems do not yet read it.
The opposite argument we hear regularly: "if the major vendors do not use it yet, why bother?". The cost-benefit is so heavily tilted toward "bother" that the question rarely survives 60 seconds of honest scrutiny.
How to write one
The structure is intentionally simple. Five elements.
Step 1: Top-level heading
The first line is a markdown H1 with the site name. For example: # The SEO Company.
Step 2: Short site description
One paragraph under the H1 describing what the site is and who it serves. Two or three sentences. Treat this as the description an AI would read aloud.
Step 3: Section headings for content categories
H2 headings for each major content category on the site. Services, Learn Hub pillars, Industries, Locations, About. For a typical small business this is four to eight sections.
Step 4: Curated URL list under each section
Under each H2, a bulleted list of the most important canonical URLs in that section, each with a short description. Format: - [Page title](https://example.com/page/): One-sentence description.
Step 5: Upload and verify
Save the file as llms.txt and upload to the document root. Confirm the file is accessible at yourdomain.com/llms.txt with a 200 status code. Re-check after any site migration.
A worked example structure
The shape we use for a Perth services business site:
- Site description. Two-sentence intro covering what the business does and where it operates.
- Services section. Each major service page with a one-sentence description.
- Learn Hub section. Each pillar page with a description, plus a short list of the most important clusters.
- Industries section. Each industry page if the site has them.
- Locations section. Each major location page with a one-line description.
- About and contact. The About page, contact page, and any About-the-business resources.
The whole file lands at 30 to 80 lines for most small businesses. Resist the urge to list every URL; the value is in curation, not completeness. The XML sitemap covers completeness; this file covers "if you only read 20 pages, read these".
llms.txt vs llms-full.txt
The proposed standard includes an optional companion file, llms-full.txt, that contains the full text content of the pages listed in llms.txt, concatenated into a single fetchable file.
When to add llms-full.txt
For documentation-heavy sites, knowledge bases, or sites where AI systems benefit from pulling a lot of context in a single fetch. The file can be large (multi-megabyte) and should be auto-generated rather than hand-written. For smaller marketing sites with 20 to 50 important pages, llms-full.txt is usually overkill.
When to skip it
For most Perth small business sites, llms.txt is enough. The pages it points to are already crawlable by AI systems; llms-full.txt only saves them a few HTTP requests. The juice is not always worth the squeeze.
Common mistakes
- Treating llms.txt as cheap insurance and adding it without overthinking adoption status.
- Curating the file to the 20 to 50 most important pages, not listing everything.
- Writing the descriptions for the AI reader: clear, factual, useful.
- Updating the file when major pages get added or removed.
- Validating that the file is accessible at the correct URL with a 200 status.
- Waiting for the major vendors to formally adopt the standard before adding the file.
- Listing every URL on the site. That is what the XML sitemap is for.
- Writing the descriptions in marketing voice rather than factual voice.
- Leaving the file stale after migration or major content updates.
- Treating llms.txt as a replacement for robots.txt or the XML sitemap.
Perth and WA context
Two observations from running llms.txt rollouts across Perth and WA clients in 2026.
Trade and services businesses get a tidy llms.txt with very little effort. The structure (services, locations, About, contact, learn resources) maps cleanly to a Perth plumbing or electrical business. The whole file is usually under 50 lines. Update cadence is low because the service pages do not change often. See trades SEO and the wider Local SEO pillar.
Larger content sites benefit more from the curation step. A Perth law firm or healthcare publisher with hundreds of articles gets the most out of llms.txt because the curation forces a decision about which 30 to 50 pages actually matter. That decision is useful for the wider content programme, not just for the AI signal. See the Content Strategy pillar.
For the wider context, the AI Search pillar covers where llms.txt sits in the broader AI visibility stack. The robots.txt chapter covers the related file that handles crawler permissions, and the XML sitemaps chapter covers the URL inventory. For an entry-level diagnostic of whether your site has the three files configured correctly, the free SEO audit includes the checks, and the website audit service includes a full review.