# Website AI-Readiness Checklist: 12 Checks With Commands

**Short answer.** AI readiness is a testable state: AI crawlers get your HTML without being blocked, find the text without running JavaScript, and can lift an answer out of it. You verify it with commands, not intuition — the status code a bot User-Agent gets, the word count in raw HTML, whether JSON-LD parses, and time to first byte.

## What AI readiness is and how it differs from SEO

 SEO answers the question "where does this page rank". AI readiness answers a different one: "can a model pull a fact out of this page and name the source". The overlap is large — both need reachable HTML and a clear structure — but the checks are different. In SEO you look at index coverage and queries; in AI readiness you look at what physically comes back in response to an HTTP request from a bot.

 The practical consequence: a site can be indexed by Google, get traffic, and still serve an AI crawler a 403 from the CDN or an empty shell with no text. Neither shows up in Search Console — it only shows up in your logs and in a `curl` response.

> AI readiness is not a score. It is a set of binary conditions: either the bot got the text or it did not. Anything you cannot check with a command and get a yes/no from does not belong on this list.

## How to use the checklist: what counts as a pass

 Every item below has the same shape — what you check, the command, what a pass looks like, what a failure looks like. Run it against one representative page of each template: home, listing, product or service page, article. Failures are almost always template-wide rather than page-specific.

 Set up a variable with a real User-Agent string first; half the checks rely on it. Take the strings only from the operator's own documentation — the URL is right there in the User-Agent after the `+` sign.

```
`UA_GPT='Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' UA_BROWSER='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36' SITE='https://example.com'`
```

 The version number in the string changes over time — when filtering logs, match the substring `GPTBot`, never an exact version.

## Block 1. Access: does your site let AI crawlers in

 The most common failure is not missing markup — it is a plain refusal at the door. And the refusal happens in three different places, each fixed in a different place.

### Item 1. robots.txt does not disallow the tokens you want

```
`curl -sS "$SITE/robots.txt" | grep -inE -A6 'user-agent:[[:space:]]*(\*|GPTBot|OAI-SearchBot|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Google-Extended|CCBot)'`
```

 **Pass:** for every token you want to admit, its group contains no `Disallow: /`. **Fail:** a `User-agent: *` group with `Disallow: /` and no permissive per-bot groups; or the file returns 404 instead of 200 (under RFC 9309 that means crawling is allowed, but you lose both control and the `Sitemap` line).

 The trap is group selection. A crawler applies **one** most specific group and ignores the rest entirely. If `GPTBot` has its own group, the `Disallow` under `User-agent: *` does not apply to it — and conversely, a carefully written wildcard block does nothing to a bot that has its own group with `Allow: /`. Full allow/deny scenarios live in the companion piece on [robots.txt and AI bots](/en/articles/robots-txt-ai-crawlers).

> robots.txt edits are not instant. OpenAI and Perplexity both document that their systems need roughly a day to pick up a changed file. Do not draw conclusions ten minutes after the edit.

### Item 2. The WAF and CDN are not returning 403 to bots

 robots.txt is an agreement; a WAF is an actual barrier. Some cloud providers ship AI-crawler blocking in their default managed rules, and the site owner never finds out. Compare two requests:

```
`for ua in "$UA_GPT" "$UA_BROWSER"; do curl -s -o /dev/null -A "$ua" \ -w 'code=%{http_code} bytes=%{size_download} ttfb=%{time_starttransfer}s\n' "$SITE/" done`
```

 **Pass:** both return 200 with comparable body sizes. **Fail:** the bot string gets 403, 429, 503, or a 200 whose body is a few kilobytes — that is a challenge page, not your content.

 **What this test does not tell you.** It cannot see IP-based rules: your address is not in the operator's ranges, so "allowed by IP" is not something you can verify from your own machine. The reverse case matters too — if a WAF lets any request through purely because the string says `GPTBot`, that is a hole, not a pass. The only real proof that a bot arrived is your logs (see [how AI crawlers read your site](/en/articles/how-ai-crawlers-read-sites)).

### Item 3. You know the price of each block

 "Block all AI bots" costs different things for different tokens. The operators document the consequences explicitly.

 TokenOperatorWhat it doesWhat you lose by disallowing it GPTBotOpenAICollects content for model trainingUse of your content in training. Does not affect ChatGPT search results OAI-SearchBotOpenAIIndexing for ChatGPT searchBeing surfaced in ChatGPT search answers ChatGPT-UserOpenAIFetches a page when a user asks for itUser-initiated, so robots.txt rules may not apply ClaudeBotAnthropicCollects content for trainingInclusion of your material in training sets Claude-SearchBotAnthropicIndexing to improve search qualityAccuracy and visibility in search answers Claude-UserAnthropicFetches a page on a user requestVisibility in user-directed web search PerplexityBotPerplexityIndexing for linked resultsAppearing in Perplexity results Perplexity-UserPerplexityFetches on a user requestDocumented as generally ignoring robots.txt Google-ExtendedGoogleControl token only; it has no User-Agent string of its ownGemini training and grounding. Does not affect Google Search inclusion or ranking Applebot-ExtendedAppleControl token layered on ApplebotUse of your material in Apple foundation-model training CCBotCommon CrawlOpen dataset many parties build onPresence in the public corpus

> Google-Extended will never appear in your logs: it has no User-Agent of its own, and the fetching is done by regular Google agents. Grepping access.log for it is wasted time.

## Block 2. Extractability: can the bot find the text

 Access granted — now, are there any words in the response? This is where more than half of modern client-rendered sites break.

### Item 4. The main text exists in the raw HTML

 Look at the raw server response, not the Elements panel: the inspector shows the DOM after scripts have run.

```
`curl -sL -A "$UA_GPT" "$SITE/page" \ | python3 -c "import sys,re,html t=sys.stdin.read() t=re.sub(r'(?is)<script.*?</script>|<style.*?</style>|<!--.*?-->',' ',t) t=re.sub(r'(?s)<[^>]+>',' ',t) t=re.sub(r'\s+',' ',html.unescape(t)).strip() print('words:',len(t.split())) print(t[:400])"`
```

 **Pass:** the word count is in the same ballpark as what you see on the page, and the first 400 characters are real page copy, not just the menu. **Fail:** 30–80 words and nothing but navigation and footer — the body is assembled on the client and the bot will not see it.

 A useful rule of thumb: if the page visually holds 800 words and the command reports under 200, the content is client-side. Techniques for getting text back into the HTML are covered in [content extractability](/en/articles/ai-content-extractability-guide).

### Item 5. A direct answer sits in the first paragraph

 A model assembles answers out of passages. A passage that opens with a definition or a conclusion gets quoted far more readily than a warm-up paragraph. The check is simple — read the first 60 words and ask whether they alone answer the question in the heading.

```
`curl -sL "$SITE/page" \ | python3 -c "import sys,re,html t=sys.stdin.read() m=re.search(r'(?is)<h1[^>]*>(.*?)</h1>(.*?)<h2', t) if not m: print('no h1 or h2 found'); raise SystemExit f=lambda s: re.sub(r'\s+',' ',html.unescape(re.sub(r'<[^>]+>',' ',s))).strip() print('H1:', f(m.group(1))) print('LEAD:', ' '.join(f(m.group(2)).split()[:60]))"`
```

 **Pass:** a 40–80 word lead containing a direct answer and the key entities. **Fail:** no text at all between `h1` and the first `h2` (straight into an image, a subscribe box, breadcrumbs), or a lead that does not answer the heading.

### Item 6. Markup is semantic, not a wall of divs

```
`curl -sL "$SITE/page" | grep -oE '</?(h1|h2|h3|ul|ol|table|article|main|nav)\b' \ | sort | uniq -c | sort -rn`
```

 **Pass:** exactly one `h1`, several `h2`, lists present, and at least one `main` or `article`. **Fail:** zero `h2`, two or more `h1`, headings faked with large text inside a `div`.

> Headings are not cosmetics. They define passage boundaries: where there are no `h2` elements, the page is one long indivisible block to a machine, and cutting a precise answer out of it is harder.

### Item 7. Text is not trapped inside images

 A screenshot of a pricing table, an infographic full of numbers, a labelled diagram — to a model that is blank space. Run the command from item 4 and check whether the numbers and terms that live on your images survive in the text. If a key figure exists only inside a `.png`, repeat it as text or as a table beside the image.

## Block 3. Structured data: what actually gets read

### Item 8. JSON-LD is in the raw HTML and it parses

 The typical failure is not missing markup — it is markup injected by a tag manager or a theme script. Visible in DevTools, absent from the server response.

```
`curl -sL -A "$UA_GPT" "$SITE/page" \ | python3 -c "import sys,re,json,html raw=sys.stdin.read() blocks=re.findall(r'(?is)<script[^>]*ld\+json[^>]*>(.*?)</script>', raw) print('JSON-LD blocks in raw HTML:', len(blocks)) for i,b in enumerate(blocks,1): try: d=json.loads(html.unescape(b)); d=d if isinstance(d,list) else [d] print(i,'ok @type =', [x.get('@type') for x in d]) except Exception as e: print(i,'DOES NOT PARSE:', e)"`
```

 **Pass:** at least one block, every block parses, and `@type` matches what the page actually contains. **Fail:** zero blocks in raw HTML while the browser shows markup; a parse error from a stray comma; a `FAQPage` on a page with no visible questions and answers.

 Which types are worth the effort is covered separately in [Schema.org for AI search](/en/articles/structured-data-for-ai-search). All that matters here is the fact: the markup must be in the server response and must not contradict the visible text.

### Item 9. Metadata is neither empty nor duplicated

```
`curl -sL "$SITE/page" | grep -oiE '<title>[^<]*|<meta[^>]+(description|robots|canonical)[^>]*' | head`
```

 **Pass:** a non-empty `title`, a description, a canonical URL, and no `noindex` on a page you want visible. **Fail:** a `noindex` left over from staging — the most expensive typo on this list.

## Block 4. llms.txt: what it does and what it does not

 The `/llms.txt` file is a community-proposed format: a short site description plus a list of priority material in Markdown. It is cheap, it does no harm, and it works nicely as a machine-readable table of contents for your own integrations.

```
`# Site name > One sentence: what the site is useful for and to whom. ## Core - [AI-Readiness Checklist](/articles/ai-readiness-checklist): what to check and with which command - [How AI crawlers read your site](/articles/how-ai-crawlers-read-sites): user agents, JS, logs ## Reference - [llms.txt guide](/articles/llms-txt-guide): format and common mistakes`
```

 **Pass:** the file returns 200 as `text/plain` and every link inside it is alive.

```
`curl -sI "$SITE/llms.txt" | head -n 3 curl -s "$SITE/llms.txt" | grep -oE '\]\(/[^)]+\)' | tr -d '](' | while read p; do printf '%s %s\n' "$(curl -s -o /dev/null -w '%{http_code}' "$SITE$p")" "$p" done`
```

 **The honest caveat.** No major AI-search operator has publicly confirmed that it consumes `llms.txt` during crawling. Treating it as a route into AI answers is unfounded. It is a nice-to-have index, not a substitute for reachable HTML and a sitemap. Format and pitfalls are in the [llms.txt guide](/en/articles/llms-txt-guide); you can validate your own file with the [llms.txt checker](/en/llms-txt).

## Block 5. Response speed and payload weight

### Item 10. Time to first byte and HTML size are sane

 A crawler does not wait forever and does not download an endless document. Google documents its limits openly: when crawling for Search it takes the first couple of megabytes of a supported file type (the docs state 2 MB, and 64 MB for PDF), the limit applies to **uncompressed** data, and every referenced resource is fetched separately under the same cap. Other operators publish no numbers — so measure and keep headroom.

```
`curl -sL -o /tmp/page.html -A "$UA_GPT" \ -w 'code=%{http_code}\ndns=%{time_namelookup}\nconnect=%{time_connect}\nttfb=%{time_starttransfer}\ntotal=%{time_total}\nredirects=%{num_redirects}\n' "$SITE/page" wc -c /tmp/page.html`
```

 **Pass:** `ttfb` under roughly 0.8 s on a cold request, `num_redirects` of 0 or 1, uncompressed HTML in the hundreds of kilobytes rather than megabytes. **Fail:** multi-second TTFB, a chain of three or more redirects, a document bloated to several megabytes by inlined data.

 curl -w fieldWhat it showsPass benchmark `%{http_code}`Final status code200 `%{time_namelookup}`Name resolution< 0.1 s `%{time_connect}`TCP connect< 0.2 s `%{time_starttransfer}`Time to first byte< 0.8 s `%{num_redirects}`Number of hops0–1 `%{size_download}`Body sizeHundreds of KB, not megabytes Measure from an external node rather than your laptop on a home connection: a [speed test](/en/speed) and a [response-header analysis](/en/http-headers) give the same picture without your ISP in the way. Compression has its own write-up — [gzip and Brotli](/en/articles/gzip-brotli-compression).

### Item 11. The site answers consistently, not usually

 Crawls happen at moments you do not control. If the site returns 502 for five minutes once a day, the odds of landing in that window are not zero, and the next visit may be weeks away. Practical criterion: over the last 30 days the share of 5xx responses on public pages is close to zero. That is not an AI-specific concern — it is ordinary [uptime monitoring](/en/monitoring).

> One 503 during a crawl costs more than missing markup. Markup gets picked up on the next visit; a page that did not exist simply does not exist.

## Block 6. Discoverability: how the bot learns the page exists

### Item 12. Sitemap and internal links are in order

 Crawling starts with links. A page that no ordinary `<a href>` points to and that is missing from the sitemap gets discovered only by accident.

```
`curl -s "$SITE/robots.txt" | grep -i '^sitemap:' curl -s "$SITE/sitemap.xml" | grep -c '<loc>'`
```

 **Pass:** robots.txt carries a `Sitemap:` line, the sitemap returns 200 and contains the expected number of URLs. **Fail:** a sitemap listing `http://` URLs on an HTTPS site, entries that redirect, or pages marked `noindex`. Navigation built entirely in JavaScript with no real links is the same failure: those transitions do not exist for a bot. Details in the [sitemap.xml guide](/en/articles/sitemap-xml-guide); dead URLs are found by the [broken link checker](/en/broken-links).

## What does NOT help: six myths about AI readiness

 Half the work is not doing the pointless half. These are the recurring time sinks.

 - **"Added llms.txt, therefore I am in AI answers."** No. No major operator has publicly confirmed reading the file during crawling. A good habit, not a channel.
 - **"Blocking GPTBot protects my content from AI."** GPTBot covers training. ChatGPT search runs on a separate token, and user-initiated fetchers (ChatGPT-User, Perplexity-User, Claude-User) are triggered by a human — per the operators' own documentation, robots.txt rules may not apply to them.
 - **"I can see Google-Extended in my logs."** It has no User-Agent string of its own; it is purely a robots.txt control token. And it affects neither Google Search inclusion nor ranking.
 - **"More markup is better."** Markup that contradicts the visible page violates the guidelines and risks manual action. A `FAQPage` with no visible FAQ is worse than no `FAQPage`.
 - **"There are special AI meta tags."** Invented directives such as `ai-content` do not exist. The real mechanisms are robots.txt tokens, `noindex` and `nosnippet` — Apple explicitly documents `nosnippet` as the opt-out from broad world-knowledge answers.
 - **"Denser keywords, more citations."** What gets extracted is the passage that answers the question. Keyword density does not move that and hurts readability.

## Troubleshooting: symptom, cause, check, fix

 SymptomLikely causeHow to checkFix Bot string gets 403, browser string gets 200A WAF rule or a CDN "AI scrapers" ruleset`curl -A` with both stringsAllow the tokens you want by User-Agent together with the operator's published IP list 200, but only 40 words in the bodyContent is client-renderedWord count from item 4Server-side rendering or prerendering of key templates Markup visible in DevTools, absent from `curl`JSON-LD injected by a tag managerScript from item 8Emit JSON-LD from the server in the raw HTML The page exists but is never requestedNo inbound links and no sitemap entry`grep -c <loc>` on the sitemapAdd it to the sitemap and link to it with real anchors Many crawler requests, little valueFaceted filters and sort parameters are being crawledGroup log lines by pathDisallow parameter URLs, keep canonical ones Everything is "correct" yet nothing is citedNo original data: the page restates what a hundred others sayManual comparison against the results pageAdd your own measurements, numbers and worked cases

> The order is strict: access first, extractability second, markup and llms.txt last. Structured data on a page that returns 403 to the crawler buys you nothing.

## How to check everything at once

 Manual commands give a precise verdict for one page. For a site-wide picture, run the prepared checks:

 - [AI readiness check](/en/ai-check) — crawler access, text in the HTML, structured data and llms.txt in one pass.
 - [llms.txt validator](/en/llms-txt) — file syntax and whether the links inside are alive.
 - [robots.txt checker](/en/robots-checker) — which group applies to a given bot and a given URL.
 - [Schema.org checker](/en/schema-checker) — which types are present in the raw HTML and whether they parse.
 - [Header analysis](/en/http-headers) and [speed test](/en/speed) — status codes, redirects, TTFB.
 - [SEO audit](/en/seo-audit) — the shared foundation: titles, canonicals, duplicates.

 Related reading: [how AI crawlers read your site](/en/articles/how-ai-crawlers-read-sites) for crawl mechanics and log analysis, [GEO](/en/articles/geo-generative-engine-optimization) and [how to appear in AI answers](/en/articles/how-to-appear-in-ai-answers) for the content side, and [AI Overviews](/en/articles/google-ai-overviews-guide), [Perplexity](/en/articles/perplexity-seo-optimization), [Bing Copilot](/en/articles/bing-copilot-optimization) and [Yandex Neuro](/en/articles/yandex-neuro-optimization) for platform specifics.

## FAQ

### Is AI readiness the same as SEO?

 No. SEO targets a position in the results; AI readiness targets whether a model can extract a fact and attribute it. The foundation is shared — reachable HTML, structure, speed — but the checks differ: SEO reads webmaster reports, this reads the server's response to a bot request.

### How many of the twelve items are critical?

 The first five: robots.txt access, no WAF block, a deliberate choice of tokens, text present in the raw HTML, and a direct answer at the top. The rest amplify the result but do nothing on their own.

### Do I need llms.txt if nobody has confirmed using it?

 It is optional. Do it if it costs five minutes and a machine-readable index is useful to you. Skip it if it is delaying server-side rendering — the priorities are not comparable.

### How often should I re-run the checks?

 The full pass after any release that touches templates, the CDN or robots.txt. The quick pair — status code for a bot string and word count in the HTML — can go on continuous monitoring, so a regression shows up the same day.

### Can I allow search but block training?

 Yes. OpenAI and Anthropic separate the tokens: the training crawler and the search crawler are configured independently. Remember the user-initiated fetchers, though — they are triggered by a person, and robots.txt rules may not apply to them.

### Does serving bots a different page version help?

 No, that is cloaking: varying content by User-Agent breaks search-engine guidelines and invites manual action. The right path is one HTML document, equally complete for everyone.

### What if the site is on a builder and server-side rendering is out of reach?

 Run item 4 first: plenty of platforms do emit the text in HTML, and there is no problem to solve. If the text really is absent, the one available lever is moving key material — articles, descriptions, answers — onto pages the platform serves statically.

## Checklist

 - robots.txt returns 200; the tokens you want are not under `Disallow: /`; the one-most-specific-group rule is accounted for.
 - A bot User-Agent gets 200 with a full body, not 403 or a CDN challenge.
 - For every token you have decided deliberately what a block would cost.
 - The raw HTML contains the main text: word count matches what is visible.
 - A 40–80 word direct answer sits between `h1` and the first `h2`.
 - One `h1`, several `h2`, lists and tables present.
 - Meaningful numbers are duplicated as text rather than living only inside images.
 - JSON-LD is in the server response, parses, and does not contradict the visible text.
 - No stray `noindex`; `title`, description and canonical are filled in.
 - llms.txt, if present, is served as `text/plain` with working links.
 - TTFB under roughly 0.8 s, 0–1 redirects, uncompressed HTML not in megabytes.
 - The page is in sitemap.xml and real links point to it.

 [Check your site's AI readiness →](/en/ai-check)

---

Source: https://enterno.io/en/articles/ai-readiness-checklist
