Short answer. Content negotiation is a mechanism where the server returns a different format of the same page depending on the Accept header. For AI agents, a site can return clean markdown instead of heavy HTML when a request arrives with Accept: text/markdown or via a /page.md URL. This saves model tokens and removes the noise from navigation, ads and scripts.
Why AI agents want markdown
When an LLM or AI crawler fetches a normal HTML page, it has to wade through tags, inline scripts, styles and trackers. The model spends tokens and sometimes misreads the structure. Markdown is semantically clean text: headings, lists, code and links with no visual clutter.
- Fewer tokens — cheaper and faster processing.
- Cleaner structure — the model understands the content hierarchy more precisely.
- Better citability — clean passages are easier to extract and quote.
Content negotiation does not serve people and bots different content to manipulate rankings — it is a legitimate delivery of the same content in the format the recipient prefers. The text is identical; only the markup changes.
Two ways to serve markdown
In practice two compatible approaches are used: the Accept header and an explicit URL with a .md extension.
The first is via an HTTP header. The agent states that it prefers markdown:
curl -H "Accept: text/markdown" https://enterno.io/If the server supports negotiation, it returns a markdown representation of the homepage (for example, the contents of llms.txt) instead of HTML.
The second is a .md suffix on the article URL:
curl https://enterno.io/articles/robots-txt-guide.mdEnterno.io serves a clean markdown version of every article at /articles/<slug>.md, and on a GET / with Accept: text/markdown it returns llms.txt. Both mechanisms work at the same time.
How the server picks a format
The logic is simple: check the Accept header or the URL extension, and if markdown is requested, return the text representation with the right Content-Type.
# Pseudocode for the server logic
if request.path.endswith('.md'):
return serve_markdown(slug)
if 'text/markdown' in request.headers.get('Accept', ''):
return serve_markdown(current_page)
return serve_html(current_page)It's important to return Content-Type: text/markdown; charset=utf-8, otherwise the client may treat the response as plain text.
Table: formats and their recipients
| Request | Response | Who needs it |
|---|---|---|
| normal browser (Accept: text/html) | HTML | People |
| Accept: text/markdown | Markdown | AI agents |
| URL /page.md | Markdown | AI agents, developers |
| llms.txt | Content map | LLMs |
Additional signals for agents
Markdown delivery pairs well with other AI signals: .well-known/agent-card.json and an MCP server card describe a site's capabilities, while a Link: header on every page can point to llms.txt and the API документацию. Together they form a complete "interface for agents".
Don't serve markdown only to bots while hiding it from users — that's a gray area. The best setup: the same content is available to people (HTML) and agents (markdown) under predictable rules.
How to validate
Send a curl with the Accept: text/markdown header and confirm markdown comes back with the correct Content-Type. Then try the URL with the .md suffix. To assess content negotiation, llms.txt and structured data together, use a free AI-readiness checker.
FAQ
Isn't this cloaking?
No, as long as the content is identical. Cloaking shows bots different content from users to manipulate results. Content negotiation serves the same content in a different format.
Do I need both .md and the Accept header?
One is enough, but both together are more robust: some agents prefer an explicit URL, others a header.
Does this affect classic SEO?
Not directly. It's about AI search and citability. Core technical SEO still matters — see the SEO audit checklist.
Which Content-Type should I use?
text/markdown; charset=utf-8. That's the standard MIME type for markdown.
Is this related to llms.txt?
Yes, they complement each other. llms.txt provides a content map; content negotiation delivers a clean format of a specific page. See the llms.txt guide.