llms.txt: What It Is, How to Create and Validate It

Short answer. llms.txt is a markdown file at your site root (/llms.txt) holding a curated table of contents: a project heading, a short summary, and lists of your most important pages with a one-line explanation each. It is a community proposal, not a ratified standard — nobody is obliged to read it. The bet is cheap: an hour of work, zero risk, and it forces you to write clean descriptions.

What llms.txt actually is

A normal web page is navigation, banners, a footer, scripts, a cookie notice — and somewhere in the middle, the useful text. Everything else is noise a machine has to filter out, spending tokens and time on it. llms.txt proposes a different approach: you write down, once and by hand, what you have that matters and where to go for the answer to a specific question.

The key word is curated. This is not a dump of every URL and not a second sitemap. It is a table of contents you selected: two or three dozen pages, each with a human explanation of why it exists.

A useful analogy. sitemap.xml is a phone book — complete, machine-oriented, no explanations. llms.txt is an annotated table of contents — short, readable by both a person and a model, and it makes the shape of the site obvious at a glance.

• What the file gives you. Context efficiency when a site is being parsed, priority control ("look here first"), and one place where page descriptions are phrased by you rather than scraped from the first paragraph.
• What it does not do. It has no effect on classic search ranking. It neither grants nor denies bot access. It does not guarantee a citation. It replaces none of the existing files.

llms.txt vs robots.txt vs sitemap.xml

All three files live in the same place — the site root — and get confused constantly. The distinction is simple: robots.txt answers "where am I allowed to go", sitemap.xml answers "which addresses exist", and llms.txt answers "what matters here and what is it about".

File	Purpose	Format	Who reads it	Required
robots.txt	Crawl permissions: which paths are open to which bots	Plain directives — User-agent / Disallow / Allow	Search robots, AI crawlers, scanners	No, but honoured de facto by every serious bot
sitemap.xml	Machine list of every indexable URL with change dates	XML following the sitemaps.org schema	Search engines	No, but an established practice with real support
llms.txt	Curated, human-readable table of contents with a note per link	Markdown: one #, a summary block, ## sections with lists	LLM agents, AI scanners, developer tooling	No. A community proposal; support is not guaranteed
llms-full.txt	Expanded variant: the full text of your material concatenated into one file	Markdown, one large document	The same agents, when the whole corpus is wanted at once	No, and most sites do not need it

Two classic mistakes follow from this. First, trying to "hide" content via llms.txt: the file is not an access directive, and leaving a page out forbids nothing. Second, writing a beautiful llms.txt while AI crawlers still hit Disallow: / in robots.txt or a bot challenge at the edge. Then there is simply nobody to read it.

Order of operations: access first, meaning second. If an AI crawler is blocked in robots.txt, at the WAF or by bot protection, llms.txt changes nothing. Check access with /robots-checker, sort out user agents in robots.txt and AI crawlers — and only then work on the table of contents.

Does llms.txt actually work? An honest status

This is the question that matters, and it deserves a straight answer without marketing.

Status of the specification

llms.txt is a community proposal published as an open specification. It did not go through a standards process, it is not an RFC, it is part of no ratified standard, and it has no normative reference implementation. Formally it is a convention that each site joins voluntarily.

Support from assistants

Public commitments of the form "our crawler reads /llms.txt and uses it when answering" are generally absent from the crawler documentation of major AI platforms. Individual agents can change behaviour without announcement, so any claim that "these specific assistants officially support the format" deserves scepticism and a check against primary sources — the crawler documentation itself.

You will almost certainly see requests for /llms.txt in your access logs. But the user agent usually reveals SEO scanners, AI-readiness checkers, developer scripts and research bots — not necessarily an assistant answering a user right now.

How to treat it in practice

• As a cheap bet. An hour of work, no risk, a non-zero possible payoff. That is a reasonable ratio for a piece of infrastructure plumbing.
• Not as a guarantee. You cannot promise a client "appearing in AI answers" because of llms.txt. There is no verifiable basis for that promise.
• The side benefit always pays. To write the file you must articulate what you have that is valuable and how to describe each page in one line. Those same lines then feed title, description, intros and announcements. That work is useful whether or not anyone ever reads the file.
• As part of a set, not the whole answer. One file solves nothing on its own. It only makes sense alongside open crawler access, clean markup and extractable text — see content extractability for AI.

The verifiability rule. Exactly three things about llms.txt are measurable: the file returns 200, its structure is valid, and its links are alive. Everything else is a probabilistic bet. If someone sells you "guaranteed citations", ask for the measurement methodology.

File format: what llms.txt is made of

The format is deliberately primitive — markdown you can read with your eyes. The block order is fixed.

1. A top-level heading — # Project name. Exactly one per file, on the first line. This is the only strictly required element.
2. A short summary block — a blockquote line > … right after the heading. One to three sentences: what the project is, who it is for, why it is useful. Formally optional, but without it the file loses half its point.
3. Free markdown prose — paragraphs and lists with no headings. Details go here: which languages the content exists in, what counts as the source of truth, what the constraints are. Optional.
4. Topical sections — ## Section name, each containing a bullet list of links in the form - [Name](absolute URL): explanation. Section names are free-form, in any language.
5. The ## Optional section — special, and always last. It means: these links may be skipped when context is tight. Everything nice-to-have goes here — blog, archive, changelog, about page.

Why ## Optional must not be translated

It is a reserved name, not an ordinary heading. The section is recognised by the literal word Optional. Translate it into your language and it reads as just another topical section, losing the entire "you may skip this" signal. Every other section heading can be written however you like.

llms-full.txt: when the expanded version is justified

If llms.txt is the table of contents, llms-full.txt is the book itself: the full text of your material concatenated into a single markdown document so an agent does not have to follow links. It is a fundamentally different file and the two should not be conflated.

• When it is justified. Product documentation, a knowledge base, an API reference, a rulebook — material that is meaningfully read end to end and fits in a sane size.
• When it is not. A store with tens of thousands of product pages, a news feed, a forum. The file balloons to tens of megabytes and nobody will fetch it.
• Practical ceiling. Keep it within single-digit megabytes. Beyond that, split by cluster — llms-docs.txt, llms-glossary.txt, llms-how-to.txt — and link to them from the main llms.txt like any other resource.
• Common mistake. Building llms-full.txt as "a list of every URL". A complete address list is what sitemap.xml is for; duplicating it in another format is pointless. "Full" here means full text, not a full list.

Three complete examples

Below are three finished files you can use as a skeleton. Pay attention to the notes after the colon — that is what a reader decides on when choosing whether to follow a link.

SaaS with documentation

The canonical case the format was designed for: a lot of structured documentation and a user with a specific technical question.

# Acme Deploy

> Acme Deploy is a continuous delivery service: build from Git, deploy to Kubernetes, roll back in one click. Works with GitHub, GitLab and Bitbucket. Free tier covers one project.

Documentation is maintained in English and Russian; the English version is authoritative. Every configuration example is verified against the current CLI release. Billing and quota questions live under "Plans and limits".

## Getting started

- [Quickstart](https://example.com/docs/quickstart): first deploy in 10 minutes, from connecting a repository to a working URL
- [Install the CLI](https://example.com/docs/cli-install): packages for Linux, macOS and Windows, release signature verification
- [Connect a repository](https://example.com/docs/connect-repo): access scopes, webhooks, self-hosted GitLab

## Configuration

- [acme.yaml reference](https://example.com/docs/config-reference): every field, defaults, validation schema
- [Environment variables and secrets](https://example.com/docs/secrets): storage, rotation, access from build steps
- [Deployment strategies](https://example.com/docs/strategies): rolling, blue-green, canary and when each fits
- [Rollback](https://example.com/docs/rollback): automatic rollback on failing health checks and manual rollback to any prior build

## API

- [Authentication](https://example.com/api/auth): tokens, lifetimes, scope restrictions
- [REST API reference](https://example.com/api/reference): endpoints, response codes, rate limits
- [Webhooks](https://example.com/api/webhooks): deploy events, request signing, delivery retries

## Troubleshooting

- [Build fails at the install step](https://example.com/docs/troubleshooting/install): cache, private registries, Node version mismatch
- [Deploy stuck in pending](https://example.com/docs/troubleshooting/pending): cluster quotas, image pull secrets, taints
- [Error codes](https://example.com/docs/error-codes): every ACD-1234 style code explained with the fix

## Plans and limits

- [Plans](https://example.com/pricing): what the free tier includes, how build minutes are counted
- [Limits](https://example.com/docs/limits): concurrent builds, artifact size, API request rate

## Optional

- [Blog](https://example.com/blog): releases and incident write-ups
- [Status page](https://status.example.com): current availability and incident history
- [Changelog](https://example.com/changelog): changes by version

Online store

The main mistake stores make is trying to list products. There are thousands of them, and sitemap.xml plus a product feed already cover that. What belongs in llms.txt is catalogue sections and, above all, the reference pages — those are what people ask an assistant about.

# Northern Light — lighting and components

> Online lighting store: 4,200 items, nationwide delivery, 24-month warranty. Trading since 2014, showroom in the city centre, we invoice business customers.

Individual product pages are not listed here — the full address list is in sitemap.xml. This file covers catalogue sections and the reference material that answers the questions buyers actually ask.

## Catalogue

- [Ceiling lighting](https://example.com/catalog/ceiling): chandeliers, spots, track systems, recessed fixtures
- [Wall lighting](https://example.com/catalog/wall): sconces, picture lights, outdoor fixtures
- [Lamps and light sources](https://example.com/catalog/lamps): E14, E27, GU10, GU5.3 bases — wattage and colour temperature
- [Components](https://example.com/catalog/parts): drivers, dimmers, track rails, mounting hardware

## How to choose

- [How many lumens a room needs](https://example.com/guides/lumens): illuminance calculation by area and room purpose, with a reference table
- [Colour temperature: 2700K, 4000K, 6500K](https://example.com/guides/color-temperature): where each belongs and how not to ruin a bedroom
- [Dimmer compatibility](https://example.com/guides/dimmers): which LED lamps dim, why light flickers and how to fix it
- [IP rating for bathrooms and outdoors](https://example.com/guides/ip-rating): IP44, IP65 and IP67 explained, and where each is required
- [Sizing a track system](https://example.com/guides/track-systems): length, number of sections, maximum load

## For buyers

- [Delivery and pickup](https://example.com/delivery): times and cost by region, free delivery threshold
- [Payment methods](https://example.com/payment): card, business invoice, instalments
- [Returns and exchanges](https://example.com/returns): deadlines, procedure, what cannot be returned
- [Warranty](https://example.com/warranty): terms, what counts as a warranty case, how to file a claim

## Optional

- [About us](https://example.com/about): history, showroom address, company details
- [Customer reviews](https://example.com/reviews): verified-purchase reviews

Blog or publication

For a blog two blocks matter most: who you are (a trust signal) and what your anchor material is. Listing every post is unnecessary — give the sections plus a dozen of your strongest pieces.

# The Observer's Notebook

> An independent blog about urban transport infrastructure: analysis of reforms, work with open data, interviews with planners. Written by a transport engineer, publishing since 2019.

Articles are published under a real name and every text cites its data sources. Corrections are marked with a note at the end of the article.

## About

- [About the author](https://example.net/about): background, experience, conflict-of-interest statement
- [How we handle data](https://example.net/method): which sources we use, how figures are verified, corrections policy

## Anchor articles

- [Counting passenger flow without turnstiles](https://example.net/passenger-flow): methodology, common errors, a ready-made calculation sheet
- [Bus lanes: data from five cities](https://example.net/bus-lanes): before-and-after comparison with the raw datasets
- [Why interchange hubs fail](https://example.net/interchange): five recurring planning mistakes on real examples
- [Where to find open transport data](https://example.net/open-data-sources): portals, formats, update frequency

## Sections

- [Transport reforms](https://example.net/tag/reforms): analysis of specific city reforms
- [Open data](https://example.net/tag/open-data): working with datasets and visualisation
- [Interviews](https://example.net/tag/interviews): conversations with planners and urbanists

## Optional

- [Archive by year](https://example.net/archive): everything published since 2019
- [Newsletter](https://example.net/newsletter): a digest every two weeks

What to include and what to leave out

Selection is the only hard part of the job. Everything else is mechanics.

Include

• Pages that answer questions. Guides, references, calculators, comparisons, error explanations, FAQs. Conversations with an assistant start with a question.
• Pages that state facts about you. Delivery terms, pricing, warranty, contacts, company details, opening hours. This is the most common category of questions about a business — and the category where an assistant is most likely to invent an answer if the fact is missing.
• Anchor pages per topic. One or two per theme, not the whole list.
• Pages you are proud of. Anything in this file may be paraphrased and cited.

Leave out

• Everything from the sitemap. That is precisely what the format exists to avoid. A URL dump without explanations helps nobody.
• Utility pages. Cart, checkout, account, on-site search, URLs with tracking or filter parameters, print views.
• Anything gated. Whatever sits behind authentication or under Disallow in robots.txt. Inviting an agent to a place you have deliberately closed is a direct contradiction.
• Duplicates and pagination. Second and third listing pages, print versions, language duplicates of the same page (give another language its own file on its own host or section).
• Pages scheduled for deletion. Anything you plan to remove soon. The link will outlive the page and turn into a 404.

On the explanations. The text after the colon is a working element, not decoration: it is what a decision to open the page is made on. "Read more", "our article", "details here" waste the line. Say which specific question the page answers and what is on it: "illuminance calculation by area with a reference table" works, "everything about lighting" does not.

On size. A sane target is dozens of links, not thousands. If the file does not fit into one screen of human attention, it will not help a machine either — the value of curation is in the exclusion.

A simple sanity check: if you are not comfortable with an assistant paraphrasing a page and citing it to a stranger, that page does not belong in llms.txt.

Where to put the file and how to serve it

Half the problems with llms.txt are not about content but about delivery. The requirements are short and strict.

• The domain root, strictly: https://example.com/llms.txt. Not /docs/llms.txt, not a subdirectory. Every subdomain gets its own file at its own root.
• Response code 200. The classic trap is a soft 404: the server returns a nicely designed "not found" HTML page with status 200. A scanner sees "the file exists" while the body is site markup. In a browser it looks like a working link.
• Content-Type: text/plain; charset=utf-8 (text/markdown; charset=utf-8 is acceptable). If it is served as text/html, some parsers will start looking for markup and find none.
• UTF-8 without a BOM. A BOM sticks to the first character and the line # Name stops being recognised as a heading.
• No protection on that path. No basic auth, no captcha, no geoblocking, no bot-protection challenge. Do not test from the browser where you are already logged in and pre-cleared.
• HTTPS and minimal hops. One redirect from http to https is fine. A three-hop chain that drops the path is not.
• Internal rewrite, not an external redirect. If the file is generated by a script, route the request to a handler with a web-server rewrite rather than answering 301 to /llms.php. An external redirect moves the content off the canonical address and adds a hop.

# nginx: a static file with a guaranteed type and charset
location = /llms.txt {
    default_type text/plain;
    charset      utf-8;
    add_header   Cache-Control "public, max-age=3600";
    try_files    /llms.txt =404;
}

# nginx: same path, content produced by a PHP handler.
# Internal rewrite — the address bar does not change, there is no redirect.
location = /llms.txt {
    rewrite ^ /llms.php last;
}

# Apache (.htaccess at the root): force the type for a ready-made file
<Files "llms.txt">
    ForceType text/plain
    Header set Cache-Control "public, max-age=3600"
</Files>

# Apache: internal rewrite to a handler — no external redirect.
# The [L] flag stops rule processing; [R] is deliberately NOT used.
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^llms\.txt$ /llms.php [L]

<?php
// llms.php — serve the file with a guaranteed status code and content type.
// Lives next to the root and is reached through the rewrite rules above.
declare(strict_types=1);

$path = __DIR__ . '/data/llms.txt';

if (!is_readable($path)) {
    http_response_code(503);
    header('Retry-After: 3600');
    header('Content-Type: text/plain; charset=utf-8');
    echo "llms.txt is temporarily unavailable\n";
    exit;
}

header('Content-Type: text/plain; charset=utf-8');
header('Content-Length: ' . (string) filesize($path));
header('Cache-Control: public, max-age=3600');
readfile($path);

Publishing on Tilda, Bitrix, WordPress, MODX and static hosting

Technically the task reduces to one thing: the file must physically sit in the public root, or a request to that root path must reach code that returns the right body with the right headers. The rest is platform specifics.

Platform	How to get the file into the root	Gotcha
Tilda	Site settings include a section for uploading arbitrary files to the domain root (the exact label depends on the panel version). Upload a ready llms.txt and republish the site.	The feature may be unavailable on entry-level plans or without a custom domain. The file is served from the builder's infrastructure — verify the status code and Content-Type by requesting your own domain, not from the panel. There is no auto-update: when the site structure changes, you re-upload by hand.
1C-Bitrix	Drop the file into the public root (DOCUMENT_ROOT) over SFTP or through the admin file manager. The web server serves an existing file before the request ever reaches urlrewrite.php.	Permissions: the file must be readable by the user PHP-FPM or Apache runs as, otherwise you get a 403. If the file is script-generated, the rule has to be added explicitly in .htaccess or the nginx config, because the built-in 404 handler returns HTML. Also confirm the composite cache is not serving a cached HTML response on that path.
WordPress	The most reliable route is uploading the file over FTP or SSH to the same directory as wp-config.php. The standard WordPress .htaccess rule only intercepts non-existent paths, so a physical file always wins.	If you build a virtual route through a hook instead, you must set the header and status 200 and then terminate execution — otherwise the engine renders a template and returns HTML. Some SEO plugins generate llms.txt on their own: make sure you have not created two sources of truth overwriting each other.
MODX	Simplest: a physical file in the root next to index.php. The native option is a resource with the alias llms, content type text/plain with a .txt extension, and an empty template field.	Leave the default template in place and you will get an HTML wrapper around your markdown. Skip the content type change and the address becomes /llms.html. The resource route also requires friendly URLs to be enabled, otherwise you end up with an ID parameter in the address.
Static hosting and CDN	Put the file in the publish directory (public/, dist/, static/) — the build tool ships it verbatim. For object storage, upload an object keyed llms.txt at the bucket root.	Object storage guesses the type from the extension: .txt usually yields text/plain but often without a charset, which breaks non-ASCII text. Set the metadata explicitly. Make sure the bundler does not hash the filename or prune it as an unused asset. Purge the CDN cache after every update.

If the platform will not let you write to the root

This happens with closed website builders, SaaS platforms and corporate portals. Options, best first:

1. A rule on the web server or reverse proxy in front of the platform. If your own nginx, Traefik or similar sits in front of the CMS, intercept /llms.txt and serve it from disk or proxy it to a separate backend. The cleanest option: the platform is not involved at all.
2. A virtual route in application code. When you have code access but no root access: register a handler for the path, set Content-Type and status 200 manually, and terminate before the framework starts rendering a template. This is exactly the "llms.txt through a script" case.
3. An edge or serverless function. If the site sits behind a CDN supporting custom functions, an edge handler can serve the file without touching the origin.
4. An external redirect — last resort. A 301 from /llms.txt to another address technically works: clients that follow redirects will reach the content. But the canonical path stops returning 200, a hop is added, and some scanners flag it. Use it only when nothing above is available, and never in place of an internal rewrite.

Generating the file: by hand, by script, on a schedule

By hand

With up to a hundred meaningful pages, write it by hand. It takes an hour or two and produces the better result: the explanations are human rather than scraped from whatever paragraph came first. Take the examples above as a skeleton, swap the links, and read every explanation out loud.

By script, from a sitemap or a database

On a large site manual assembly is unrealistic and, more importantly, goes stale fast. Two data sources:

• From sitemap.xml — universal and works without database access: take the addresses, filter by section, pull title and description from each page.
• From the database or CMS — more accurate and much faster: you already have the heading, the summary, the update date and an "important page" flag, with no HTTP fetching or HTML parsing.

#!/usr/bin/env python3
# build_llms.py — assemble llms.txt from sitemap.xml
import re
import urllib.request
import xml.etree.ElementTree as ET

SITEMAP  = "https://example.com/sitemap.xml"
NS       = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
SECTIONS = [("/docs/", "Documentation"), ("/guides/", "Guides")]
HDR      = {"User-Agent": "llms-txt-builder/1.0 (+https://example.com/)"}

def get(url):
    req = urllib.request.Request(url, headers=HDR)
    with urllib.request.urlopen(req, timeout=20) as resp:
        return resp.read().decode("utf-8", "replace")

def clean(m):
    return re.sub(r"\s+", " ", m.group(1)).strip() if m else ""

def title_desc(html):
    t = re.search(r"<title>(.*?)</title>", html, re.S)
    d = re.search(r'name="description"\s+content="(.*?)"', html, re.S)
    return clean(t), clean(d)

urls = [loc.text.strip() for loc in ET.fromstring(get(SITEMAP)).findall(".//sm:loc", NS)]

out = ["# Example", "", "> A short description: what we do and who for.", ""]
for prefix, heading in SECTIONS:
    picked = sorted(u for u in urls if prefix in u)
    if not picked:
        continue
    out += ["## " + heading, ""]
    for u in picked:
        title, desc = title_desc(get(u))
        out.append("- [%s](%s): %s" % (title or u, u, desc or "no description"))
    out.append("")

with open("llms.txt", "w", encoding="utf-8") as f:
    f.write("\n".join(out) + "\n")

Why generation must be automated

A file assembled once "to see how it goes" starts doing harm six months later: it points at deleted sections and renamed pages. You are personally sending an agent to a 404 — a worse quality signal than having no file at all.

Two places worth running the generator:

• On a schedule. Once a night is enough for most sites. A crontab entry:

# crontab -e  — daily rebuild at 04:15 with a log
15 4 * * * /usr/bin/python3 /opt/site/build_llms.py >> /var/log/llms-build.log 2>&1

# weekly link check, Sundays at 05:30
30 5 * * 0 /opt/site/check_llms_links.sh >> /var/log/llms-links.log 2>&1

• In the build pipeline. A post-deploy step guarantees the file never lags behind the site. Put the link check there too and fail the build when something breaks.

The limit of automation: descriptions pulled from meta description are almost always weaker than hand-written ones — they were written for a search snippet, not to answer a question. A workable hybrid is to write the section structure and a dozen key pages by hand, and let a script fill the long tail (glossary, error-code reference, catalogue sections).

A file full of dead links is worse than no file. If you cannot put generation on a schedule, at least set a quarterly reminder to review the content and re-check the links. "Publish once and forget" is the most common decay pattern.

How to validate llms.txt

Status code, content type, redirects

First and most important: confirm that the address really serves a text file rather than an HTML page.

# 1. One-line summary: code, type, redirect count, final address
curl -sIL -o /dev/null \
     -w 'code=%{http_code} type=%{content_type} redirects=%{num_redirects} final=%{url_effective}\n' \
     https://example.com/llms.txt

# 2. Raw headers — inspect Content-Type, Content-Length, caching
curl -sI https://example.com/llms.txt

# 3. First lines of the body: expect "# Name", not "<!DOCTYPE html>"
curl -s https://example.com/llms.txt | head -20

# 4. Is there a BOM at the start? (we expect NOT to see "efbb bf")
curl -s https://example.com/llms.txt | head -c 3 | xxd

# 5. Size in bytes — a guard against bloat
curl -s https://example.com/llms.txt | wc -c

Expected result: code=200, type=text/plain; charset=utf-8 (or text/markdown), redirects=0 when requesting the canonical host, and a first line reading # Project name.

Validating the structure

There is no normative llms.txt validator — and that is worth stating plainly. The format is not a ratified standard, so there is no reference check you can cite as mandatory. Third-party checkers verify that the file is reachable and formally well-shaped, but their verdict is one implementation's opinion, not conformance to a required specification.

A manual structure checklist:

• the first non-empty line starts with # , and there is exactly one such heading;
• immediately after it, a blockquote line > describing the project;
• sections at the second level only (## ), never ### or deeper;
• every link is an absolute https URL, not a relative path;
• every link carries an explanation after the colon;
• no tables, no nested lists, no raw HTML — flat markdown only;
• ## Optional is spelled in English and comes last.

A quick mechanical check of the first four points:

curl -s https://example.com/llms.txt | awk '
  NR==1 && $0 !~ /^# /            { print "FAIL: first line is not an H1: " $0 }
  /^# /                           { h1++ }
  /^#{3,} /                       { print "WARN: heading deeper than level 2: " $0 }
  /^- \[/ && $0 !~ /\): ./        { print "WARN: link without explanation: " $0 }
  /^- \[/ && $0 !~ /\(https:\/\// { print "WARN: link is not an absolute https URL: " $0 }
  END { if (h1 != 1) print "FAIL: number of H1 headings in file: " h1 }
'

Are all the links alive

The most valuable check of all: a file with dead links sends an agent nowhere. This one-liner extracts every URL from the markdown links and prints only the ones that did not answer 200.

curl -s https://example.com/llms.txt \
  | grep -oE 'https?://[^) ]+' \
  | sort -u \
  | while read -r u; do
      printf '%s %s\n' "$(curl -sL -o /dev/null --max-time 15 -w '%{http_code}' "$u")" "$u"
    done \
  | grep -v '^200 '

Empty output means you are fine. Every printed line is an address to fix or remove. The same script belongs in a build pipeline, failing the build whenever the output is non-empty.

What to check on enterno.io

• /llms-txt — file presence, structure parsing, sections and links.
• /http-headers — status code, Content-Type, charset and the redirect chain leading to the file.
• /robots-checker — whether AI crawlers are blocked in robots.txt.
• /ai-check — overall AI readiness: llms.txt, structured data, markdown negotiation, agent cards.
• /broken-links — broken links across the site, including the ones you listed in the file.

Common failures: symptom → cause → check → fix

It opens in the browser but a scanner says "not found"

Cause. A soft 404: the server returns a styled "not found" HTML page with status 200, or the browser is showing you a cached copy. Check. curl -sI plus curl -s … | head -5 — inspect the code and the first lines of the body. Fix. Put a physical file in the root or add an explicit server rule; confirm status 200 and that the body is markdown, not markup.

Content-Type is text/html instead of text/plain

Cause. The request reaches the CMS and is rendered through a template. Check. curl -sI https://example.com/llms.txt | grep -i content-type. Fix. A server rule with default_type (nginx) or ForceType (Apache); if a script produces the body, set the header explicitly and terminate before template rendering.

Non-ASCII characters come out garbled

Cause. No charset in Content-Type, the file is not saved as UTF-8, or there is a BOM at the start. Check. file llms.txt locally and curl -s … | head -c 3 | xxd remotely. Fix. Re-save as UTF-8 without BOM, add charset utf-8 to the config, and for object storage set the object metadata explicitly.

The file is perfect but AI crawlers never reach the site

Cause. Access is blocked in robots.txt, at the WAF, by bot protection or by geo rules. Check. /robots-checker plus user-agent greps over the access log. Fix. Make a deliberate decision about which bots you allow; details in robots.txt and AI crawlers and how AI crawlers read sites.

Some links in the file return 404

Cause. The file was built once and never refreshed while the site kept changing. Check. The link-walking one-liner above, or /broken-links. Fix. Scheduled regeneration plus a link check in the build pipeline that fails on non-empty output.

The file has ballooned to hundreds of kilobytes

Cause. The whole sitemap was dumped into llms.txt. Check. curl -s … | wc -c, and simply looking at the file. Fix. Keep a curated core of a few dozen links and move long enumerations into separate cluster files you link to.

The Optional section has been translated

Cause. The natural urge to localise everything. Check. By eye: the last section must read ## Optional. Fix. Restore the English name. It is a reserved word, not a heading for readers.

The file exists on one host but not another

Cause. The site answers on www and apex, and over http, but the file only lives on one variant. Check. Run curl -sIL against all four combinations. Fix. One canonical host: non-canonical variants 301 to it, the canonical one returns the file with status 200.

How it fits with robots.txt and structured data

llms.txt is one item in a set and solves little on its own. Its neighbours:

• robots.txt. Decides whether a crawler gets in at all. General mechanics in the robots.txt guide, specific AI bot user agents in robots.txt and AI crawlers. The Sitemap: line in robots.txt stays where it is — llms.txt does not replace it.
• Structured data. Schema.org gives facts machine-readable types: this is a product, this an organisation, this an author, this a price. Markup answers "what is this object", llms.txt answers "where do I even look". See structured data for AI search.
• Serving markdown via Accept. The logical continuation of the idea: the same material delivered to an agent as clean text without markup — content negotiation for AI.
• Agent cards under /.well-known/. Capability and integration descriptors for agents — agent cards and .well-known. This is where machine descriptors moved; the older .well-known/ai-plugin.json belonged to a plugin ecosystem that has been retired, and it should no longer be presented as a current companion to llms.txt.
• Content extractability. If the text is painted by client-side script, no table of contents will save it — content extractability for AI.
• The whole strategy. How this adds up: generative engine optimization and how to get cited by ChatGPT.

FAQ

Is llms.txt mandatory?

No. It is neither a standard nor a requirement of any platform — it is a community proposal. Its absence is not a site defect and is not penalised by search engines or assistants.

Does llms.txt actually work?

Honestly: there are no verifiable guarantees. Major AI platforms generally do not publish commitments to read the file, and agent behaviour can change without notice. The reasonable stance is to treat it as a cheap bet — an hour of work, no risk, possible upside — and to remember that the selection and description work pays off regardless of whether anyone reads the file.

Does llms.txt replace sitemap.xml or robots.txt?

Neither. robots.txt is crawl permission, sitemap.xml is a machine list of addresses for indexing, llms.txt is a curated table of meaning. All three solve different problems and coexist.

Is there an official llms.txt validator?

Not a normative one, because there is no ratified standard behind it. Third-party checkers exist, including ours: they inspect availability, status code, content type and formal structure. That is useful, but the verdict is one implementation's opinion rather than conformance to a mandatory specification.

How do I publish llms.txt on a site builder with no server access?

Most builders offer a settings section for uploading files to the domain root — that is enough, though you will be updating the file by hand. On a CMS you usually do have root access: upload through the admin file manager or over SFTP and check read permissions for the web-server user. If the root is genuinely unreachable, intercept the path at a web server or reverse proxy in front of the platform, or serve the body from a handler in application code.

Can I serve llms.txt with a redirect to a PHP file?

It will work, but it is not the best option. An external redirect (301/302) moves content off the canonical address and adds a hop, and some scanners flag it. Prefer an internal rewrite: rewrite ^ /llms.php last; in nginx, or RewriteRule ^llms\.txt$ /llms.php [L] in Apache. The address stays /llms.txt, there is no redirect, and a script produces the body. Remember to set Content-Type: text/plain; charset=utf-8 in the handler.

Does llms.txt affect search rankings?

It has no direct effect on classic search ranking — it is not addressed to search algorithms. Nothing stops search robots from reading it, but it should not be treated as a ranking factor. If visibility in search is the goal, work on ordinary technical SEO and on the pages themselves.

Implementation checklist

• The file sits at https://canonical-domain/llms.txt — the root, no subdirectories.
• curl -sIL reports code=200 and redirects=0 on the canonical host.
• Content-Type is text/plain; charset=utf-8 or text/markdown; charset=utf-8.
• Encoding is UTF-8 with no BOM at the start.
• The first line is the single # Project name heading.
• Right after it, a > blockquote summarising the project in one to three sentences.
• Sections are second level only, links are absolute, and each carries a meaningful explanation.
• The reserved section is spelled ## Optional in English and comes last.
• No utility, gated, duplicate or soon-to-be-deleted pages are listed.
• Size is dozens of links rather than the entire sitemap; heavy enumerations moved to separate files.
• llms-full.txt exists only if the content is genuinely read end to end, and stays within single-digit megabytes.
• Every link returns 200 — verified by walking them, not by eye.
• AI crawlers are not blocked in robots.txt, at the WAF or by bot protection.
• Generation runs on a schedule or in the build pipeline, and the link check fails the build on breakage.
• The CDN cache is purged after each update.
• Someone on the team owns the periodic review of the file's contents.

Check llms.txt on your site →