# robots.txt Guide: Syntax, Rules, Testing and Ready-Made Files

**In short:** robots.txt is a plain text file at the root of a host that tells crawlers which URLs they may fetch. It controls *crawling*, not *indexing*: a page blocked in robots.txt can still appear in search results — without a description. The syntax is defined in RFC 9309: `User-agent`, `Disallow`, `Allow`, plus the `Sitemap` extension.

## What robots.txt Is and How a Crawler Reads It

 robots.txt is a plain text file that must live at exactly one address: `https://example.com/robots.txt`. No other filename, no subdirectory. A crawler requests that path and looks nowhere else. Inside it is a list of `name: value` lines separated by line breaks.

 The sequence looks like this:

 - Before crawling a host, the bot issues `GET /robots.txt` against the same scheme, host and port.
 - The file is parsed and cached — typically for around a day, longer if caching headers say so.
 - For every URL the crawler picks **one** matching group of rules and evaluates the path against it.
 - If the path is disallowed, the crawler never requests the page. It knows the URL exists; it just never sees the content.

 The protocol is voluntary by design. robots.txt is not a firewall, not an access rule and not authentication. Major engines (Google, Bing, Yandex) obey it because it is in their interest. Scrapers, content harvesters and vulnerability scanners do not. So robots.txt solves exactly one problem: it saves crawl budget and keeps junk URLs out of the fetch queue.

> robots.txt is a request, not a restriction. Anything that genuinely must not be public belongs behind authentication or behind a 403/404. The file itself stays publicly readable by anyone with a browser.

 Why restrict crawling at all? Because a crawler has a budget of requests per site per unit of time. If 200,000 sorting and filtering URLs consume that budget, product pages get visited rarely. Removing junk redistributes the budget toward pages that actually need to rank.

 *

## robots.txt Syntax: Standard Directives and Everything That Is Not

 For decades robots.txt was a 1994 convention with no formal specification, so parsers disagreed on details. The protocol is now standardised in [RFC 9309, the Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html). It defines exactly three things: the `User-agent` line and the `Allow` and `Disallow` rules. Everything else is an extension with varying support.

### Parsing rules

 - One directive per line: name, colon, value.
 - **Directive names are case-insensitive.** `Disallow`, `disallow` and `DISALLOW` are the same. So are user-agent names.
 - **Path values are case-sensitive.** `/Admin/` and `/admin/` are different paths. This is the most common reason a rule "does nothing".
 - `#` starts a comment that runs to the end of the line.
 - The file must be UTF-8. Characters outside the allowed set may be discarded.
 - An empty `Disallow:` value means "nothing is disallowed". It is an allow rule, not a mistake.

### A minimal valid file

```
`# Minimal working robots.txt User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml`
```

 This allows the whole site and points at the sitemap. Strictly speaking a site with no restrictions needs no robots.txt at all, but returning a 404 there is not ideal: some audit tools flag it, and the `Sitemap` line is convenient to keep in this exact place.

### What is not part of the standard

 You will meet the directives below in other people's files and in old tutorials. None of them is in RFC 9309, and support varies widely.

 - **`Crawl-delay`** — a request to pause between fetches. Google has never supported it and silently ignores it. Yandex dropped it in favour of a crawl-rate setting in its webmaster panel. Bing honours it. As a load-control mechanism it is unreliable — fix the cause on the server instead.
 - **`Host`** — historically told Yandex which mirror was canonical. **Yandex has retired it**; the canonical host is now determined by 301 redirects and webmaster settings. Leaving it in the file does no damage but confuses whoever reads it next.
 - **`Clean-param`** — a Yandex extension that lists query parameters which do not change page content, so URLs can be consolidated. Yandex only; Google does not parse it.
 - **`Noindex:` inside robots.txt** — does nothing. No such directive exists in the protocol and no engine executes it. Use a meta tag or an HTTP header instead.
 - **`Request-rate`, `Visit-time`** — relics nobody honours today.

### Directive reference

 DirectiveSupportWhat it doesTypical mistake `User-agent`Standard, all crawlersOpens a group and names the crawler it applies toA typo in the bot name — the group applies to nobody and the bot falls back to `*` `Disallow`Standard, all crawlersBlocks URLs starting with the given prefixMissing leading slash; `/admin` instead of `/admin/`, which also blocks `/administration` `Allow`Standard (Google, Bing, Yandex)Explicitly permits a prefix inside a disallowed areaRelying on line order: precedence comes from rule length, not position `Sitemap`Extension, widely supportedPoints to an absolute sitemap URLA relative path such as `Sitemap: /sitemap.xml` — the line is ignored `Crawl-delay`Bing; not Google; retired by YandexAsks for a delay between requestsUsed as a substitute for proper caching and rate limiting `Host`Obsolete, retired by YandexUsed to declare the canonical mirrorThe file looks configured while the line does nothing `Clean-param`Yandex onlyConsolidates URLs differing by insignificant parametersExpecting Google to read it `Noindex`Not supported anywhereNothingThe page stays indexed while the owner believes otherwise

## The Group Rule: How Several User-agent Lines Share One Block

 A robots.txt file is not a flat list — it is a set of groups. A group consists of one or more **consecutive** `User-agent` lines followed by `Allow`/`Disallow` rules. The first rule line closes the list of names: the next `User-agent` line starts a new group.

```
`# One group for three crawlers: shared rules User-agent: Googlebot User-agent: Yandex User-agent: bingbot Disallow: /cart/ Disallow: /search # A separate group for everybody else User-agent: * Disallow: /cart/ Disallow: /search Disallow: /api/`
```

 Googlebot, Yandex and bingbot get two rules; everyone else gets three. Note that the `*` group **does not apply at all** to the three named bots. That is the main trap.

### A crawler picks exactly one group

 The bot looks for the group whose name matches it most specifically. Once found, it obeys that group only and ignores the `*` section entirely. The classic failure:

```
`User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /search Disallow: /*?utm_source= User-agent: Yandex Clean-param: sort&view /catalog/`
```

 The author meant to add "one setting for Yandex". What actually happened is that **all four restrictions were lifted for Yandex**: its bot found its own group, and that group contains no `Disallow` at all. The fix is to repeat the shared rules inside the named group in full.

> The rule is simple: if you create a named section for a crawler, that section must contain the **complete** rule set for it. Groups are never inherited and never merged with `*`.

### Duplicate groups and blank lines

 If the same `User-agent` appears twice, modern parsers — Google's among them — merge those groups into one. Do not rely on it, though: audit tools and older parsers behave differently.

 Blank lines are a separate story. The original 1994 convention used a blank line as the record separator, and some parsers still treat it as the end of a group. Google's current parser ignores blank lines. The practical conclusion: **never put a blank line inside a group**. Keep groups monolithic and use blank lines only between groups — then every parser reads the file the same way.

```
`# Bad: some parsers end the group at the blank line User-agent: * Disallow: /admin/ Disallow: /cart/ # Good: monolithic group, blank line only between groups User-agent: * Disallow: /admin/ Disallow: /cart/ User-agent: AhrefsBot Disallow: /`
```

## How Paths Are Matched: Wildcards and Allow Precedence

### Prefix matching

 A `Disallow` value is compared against the URL path as a **prefix** — not as a full path and not as a directory name. `Disallow: /admin` blocks `/admin`, `/admin/`, `/admin/users` and also `/administrator` and `/admin-guide.html`. If you mean a directory, add the trailing slash: `Disallow: /admin/`.

 The query string is part of the match. `Disallow: /*?sort=` catches `/catalog/?sort=price`. Scheme and host take no part in matching.

### The `*` and `$` wildcards

 - `*` matches any sequence of characters, including none. `Disallow: /*/print` catches `/catalog/print` and `/blog/2026/print`.
 - `$` anchors the end of the URL. `Disallow: /*.pdf$` blocks `/docs/manual.pdf` but leaves `/docs/manual.pdf.html` and `/file.pdf?v=2` crawlable, because characters follow `.pdf`.
 - `?`, `.` and `&` are ordinary characters here — there is no regular-expression behaviour.
 - Google, Bing and Yandex all support both wildcards.

 RuleURLResult `Disallow: /admin``/administrator/login`Blocked — the prefix matches `Disallow: /admin/``/administrator/login`Allowed `Disallow: /*.pdf$``/docs/a.pdf?v=2`Allowed — `$` requires end of string `Disallow: /*?``/catalog/?page=2`Blocked — every URL with a query string `Disallow: catalog/``/catalog/`Does not work — no leading slash `Disallow:`anyAllowed — an empty value blocks nothing

### Precedence: the longest rule wins, not the first

 The most widespread misconception is "whatever comes last wins" or "Allow always beats Disallow". The actual rule: **the most specific rule applies — the one with the longest path value**. Line order is irrelevant. If two rules are the same length, the less restrictive one wins, which means `Allow`.

```
`User-agent: * Disallow: /catalog/ Allow: /catalog/deals/`
```

 For `/catalog/deals/lamp` both rules match. `/catalog/deals/` is 15 characters, `/catalog/` is 9. `Allow` wins and the page is crawled. Swap the two lines and nothing changes.

```
`# The reverse case: Allow is shorter, so Disallow wins User-agent: * Allow: /catalog/ Disallow: /catalog/private/ # /catalog/private/doc is blocked (18 > 9)`
```

 `*` and `$` count as ordinary characters when measuring length, so rules with wildcards sometimes win unexpectedly. When in doubt, test the specific URL in a validator instead of counting characters in your head.

## The Big Misconception: robots.txt Does Not Remove a Page From the Index

 This is the point that damages real sites most often. **robots.txt blocks crawling, not indexing.** The difference matters.

 If a blocked page has inbound links, internal or external, a search engine may index the URL anyway. It never saw the content, so the result shows a bare URL or a title assembled from anchor text, plus a note along the lines of "No information is available for this page". The page is in the results. You did not remove it.

> The classic trap: a page is blocked in `robots.txt` **and** carries `<meta name="robots" content="noindex">`. The crawler cannot fetch the page, so it cannot read the `noindex`, so it never acts on it. The two "blocks" cancel each other out and the URL stays in the results for years.

### The correct removal sequence

 - **Unblock** the page in robots.txt — remove the matching `Disallow`.
 - Add `<meta name="robots" content="noindex, follow">` or the `X-Robots-Tag: noindex` header.
 - Wait for a recrawl. This takes weeks; you can nudge it from the webmaster consoles.
 - Only after the URL has left the index, block the path in robots.txt if you want to stop spending crawl budget on it.

 For more on the meta tags involved, see the guide to [title, description and meta tags](/en/articles/title-description-meta-tags). If the opposite is happening and pages will not appear at all, see [why a site is not in search results](/en/articles/website-not-in-search).

### Which tool solves which job

 GoalRight toolWhy not robots.txt Keep crawlers out of the admin area`robots.txt` plus authenticationrobots.txt saves budget; only auth restricts access Remove a page from the index`noindex` meta tag or `X-Robots-Tag`A blocked page is never fetched, so `noindex` is never read Keep a staging domain out of searchHTTP Basic Auth (401)A staging robots.txt easily ships to production and kills the live site Deal with filter and sort duplicates`canonical`, plus `Disallow` on parameters at scaleA blocked duplicate cannot pass its canonical signal — the bot never sees it Keep PDFs and documents out of the index`X-Robots-Tag: noindex` headerYou cannot put a meta tag in a binary file, and robots.txt does not deindex Hide confidential dataAuthentication, 403/404, move it out of the web rootrobots.txt is public and reads like a table of contents Allow or block AI crawlers`robots.txt` by bot name; `llms.txt` for content curationTwo different jobs: access versus presentation Reduce crawler loadServer-side rate limiting, caching, webmaster crawl-rate settingsGoogle ignores `Crawl-delay` and Yandex retired it

## How to Block a Site With robots.txt — and When That Backfires

 Blocking an entire site looks like this:

```
`# Block the whole site for every crawler User-agent: * Disallow: /`
```

 Note the difference that breaks sites:

 - `Disallow: /` — the entire site is blocked.
 - `Disallow:` — nothing is blocked.

 One character. A typo here is the most expensive typo in SEO.

### When `Disallow: /` is genuinely appropriate

 - A technical domain, a mirror or a staging subdomain that should never rank.
 - A site under construction that has never been indexed.
 - A parked domain bought for a redirect.

### Why it is a poor way to hide a site

 First, if the site is already indexed, `Disallow: /` will not remove it — see the previous section. Worse, you cut off the crawler's ability to ever read a `noindex`, so those URLs stay in the results as bare links.

 Second, a staging file with `Disallow: /` is a time bomb. It lives in the repository, somebody deploys the wrong branch, and the production site goes dark. It is usually discovered two or three weeks later, once traffic has already dropped.

 Third, a blocked staging robots.txt is public and effectively announces "there is a site here and somebody is hiding it". Scanners find that interesting.

> Close a staging environment with **authentication**, not robots.txt. Return 401 to everyone but your team: no crawler can obtain the content at all, and robots.txt stops being a single point of failure on deploy.

```
`# nginx: lock a staging domain properly server { server_name staging.example.com; auth_basic "Staging"; auth_basic_user_file /etc/nginx/.htpasswd; # belt and braces in case auth is ever removed add_header X-Robots-Tag "noindex, nofollow" always; }`
```

 After every release it pays to verify that the production robots.txt has not been replaced. The cheapest protection is content monitoring: if `Disallow: /` ever appears, you find out that day rather than a month later.

## Sitemap in robots.txt: Declaring Your Sitemap Correctly

 The `Sitemap` directive is an extension described at [sitemaps.org](https://www.sitemaps.org/protocol.html) and supported by Google, Bing and Yandex. Its rules differ from the other directives.

 - **Absolute URLs only.** `Sitemap: /sitemap.xml` is a dead line and is skipped. Write `Sitemap: https://example.com/sitemap.xml`.
 - **Multiple lines are allowed.** Declare as many sitemaps as you have: products, articles, images.
 - **The directive is not part of any group.** It is global, regardless of which `User-agent` blocks surround it. For readability, keep it in its own block at the top or bottom of the file.
 - **Scheme and host must match production.** If the site is on HTTPS and the sitemap is declared over `http://`, you are sending the crawler through a needless redirect.
 - **Never block the sitemap path.** Combining `Disallow: /*.xml$` with a `Sitemap` line is a reliable way to break your own indexing.

```
`User-agent: * Disallow: /cart/ Disallow: /search Disallow: /*?utm_source= Sitemap: https://example.com/sitemap.xml Sitemap: https://example.com/sitemap-products.xml Sitemap: https://example.com/sitemap-images.xml`
```

 With many sitemaps, declare a single sitemap index and list the rest inside it — robots.txt stays short. Sitemap index structure, priorities and `lastmod` are covered in the [sitemap.xml guide](/en/articles/sitemap-xml-guide).

## Where robots.txt Must Live and What the Server Must Return

### One host, one file

 robots.txt applies to a combination of **scheme + host + port**. Several non-obvious consequences follow:

 - `https://example.com/robots.txt` and `https://shop.example.com/robots.txt` are **different files**. A subdomain needs its own; the root file does not cover it.
 - `https://example.com` and `http://example.com` are formally different origins too. In practice a site serves the same file over both, which is what you want.
 - `https://example.com:8443/` is again a separate file.
 - A file in a subfolder does nothing: `/blog/robots.txt` is never requested.
 - Rules do not reach other hosts. If your assets sit on a CDN domain, they are governed by that domain's robots.txt.

### Status codes and how crawlers read them

 Response to `/robots.txt`Crawler behaviourRisk `200` with textThe file is parsed and appliedNormal `404` or another `4xx`Treated as "no restrictions": **everything is allowed**The whole site is crawled, service sections included `401`, `403`Treated as an absent file — usually everything allowedCommon when the entire site sits behind auth `5xx`Treated as unreachable — the crawler assumes **everything is disallowed**Crawling stops completely for the duration of the outage `3xx`Redirects are followed, but only a limited number (around five)A redirect chain or loop means the file is never retrieved Timeout, resetSame as `5xx`: crawling is pausedAn unstable server slows indexing

> The most underestimated incident: the site goes down and `/robots.txt` starts returning 500 along with everything else. To a search engine that reads as "the whole site is closed". If the 5xx persists, crawling halts entirely. This is why robots.txt should be served as a static file rather than rendered by the application.

 A full breakdown of status codes and their SEO consequences lives in the [HTTP status codes reference](/en/articles/http-status-codes).

### Size, encoding, Content-Type

 - **Size.** RFC 9309 requires parsers to handle at least 500 KiB. Anything beyond that may be dropped. A 2,000-line file almost always means parameters should be handled with canonicals rather than enumerated.
 - **Encoding.** UTF-8 only. Non-ASCII characters in comments are fine, but paths are safer percent-encoded, exactly as they appear in requests.
 - **BOM.** Save the file **without a BOM**. Google ignores a leading BOM, but not every parser does: three invisible bytes attach to the first line and `User-agent: *` stops being recognised. The symptom is distinctive — "the first directive is ignored, the rest work".
 - **Content-Type.** The file must be served as `text/plain`. If a framework serves it as `text/html` — typical when a controller renders robots.txt — some crawlers will not process it.
 - **Line endings.** LF and CRLF are both fine. HTML inside the file is not: no `<pre>`, no page wrapper.

## Ready-Made robots.txt for WordPress, Tilda and Bitrix

 The principle is the same for any CMS: block service areas, internal search, the cart and parameterised URLs — and **do not block CSS, JavaScript or images**. Without them the crawler cannot render the page or judge the mobile layout.

> Blocking `/wp-includes/`, `/bitrix/` or `/assets/` wholesale is not "security", it is broken rendering. The engine sees an unstyled page and evaluates it accordingly.

### robots.txt for WordPress

 WordPress has a quirk: with no physical file in the root, it serves a **virtual** robots.txt that exists only in the response. The default content blocks `/wp-admin/` while allowing `admin-ajax.php`.

 Two practical consequences. First, any physical `robots.txt` in the root fully overrides the virtual one. Second, and far nastier: the Settings → Reading → Discourage search engines from indexing this site* checkbox replaces the virtual file with `User-agent: *` and `Disallow: /`. It gets ticked during development and forgotten. If a site suddenly disappears from search, check that box first.

```
`# robots.txt for WordPress User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-login.php Disallow: /xmlrpc.php Disallow: /?s= Disallow: /search/ Disallow: /*?replytocom= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?fbclid= Disallow: /*?gclid= # Keep CSS, JS and media crawlable Allow: /wp-content/uploads/ Allow: /wp-includes/js/ Allow: /*.css$ Allow: /*.js$ Sitemap: https://example.com/wp-sitemap.xml`
```

 What is deliberately absent: blanket blocks on `/wp-content/plugins/` and `/wp-includes/`. Plugins ship front-end CSS and JS, and blocking them breaks rendering. Tags, archives and pagination are better handled with meta tags than with robots.txt — otherwise the crawler sees neither `noindex` nor `canonical`.

### robots.txt for a Tilda site

 Tilda generates robots.txt on its side and serves it automatically once the site is published to your own domain. The content can be overridden in the site settings, where the platform provides a field for your own version.

 - Tilda styles and scripts load from **its own static domains**. Your robots.txt has no effect on them — and you neither need nor can block them.
 - A site published to the platform's service subdomain is a separate host with its own robots.txt that you do not control. Keep the preview out of search by not linking to it and not verifying it in webmaster consoles.
 - The main things worth adding by hand are the `Sitemap` line and blocks for utility pages such as thank-you screens.

```
`# robots.txt for a Tilda site User-agent: * Disallow: /thanks Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Sitemap: https://example.com/sitemap.xml`
```

### robots.txt for Bitrix

 Bitrix generates the file from its SEO module, and the default output blocks `/bitrix/` entirely. That is exactly where manual editing is needed: `/bitrix/` holds `js`, `templates` and compiled CSS caches — precisely what the crawler needs in order to render.

```
`# robots.txt for Bitrix User-agent: * Disallow: /bitrix/ Disallow: /local/ Disallow: /auth/ Disallow: /personal/ Disallow: /cart/ Disallow: /search/ Disallow: /upload/ Disallow: /*PAGEN_ Disallow: /*?print= Disallow: /*?set_filter= Disallow: /*?clear_cache= Disallow: /*?utm_source= # Give the crawler the layout and scripts back Allow: /bitrix/js/ Allow: /bitrix/templates/ Allow: /bitrix/cache/css/ Allow: /bitrix/cache/js/ Allow: /local/templates/ Allow: /upload/iblock/ Allow: /*.css$ Allow: /*.js$ Sitemap: https://example.com/sitemap.xml`
```

 About `/upload/`: never block it wholesale if it stores product photography — the site then vanishes from image search. That is why `/upload/iblock/` is allowed back in the example above.

 Faceted filters are the classic Bitrix problem: they generate a combinatorial explosion of URLs. Blocking them via `Disallow: /*?set_filter=` is reasonable, but filter pages must also carry a correct `canonical`, and genuinely valuable filter combinations are better promoted to their own clean-URL landing pages and left crawlable.

## Separate Sections for Google and Yandex: When You Actually Need Them

 By default you do not. A single `User-agent: *` group covers every engine, and the shorter the file, the fewer mistakes it contains. Named sections are justified in four cases.

 - **You need `Clean-param`.** Only Yandex parses it, so it can physically live only in a Yandex group.
 - **The rules genuinely differ.** For example, one engine gets a set of images another does not, or a section is open to one and closed to the other.
 - **AI crawler control.** Google separates its model-training agent from its search crawler (`Google-Extended`), and Yandex likewise runs a separate additional crawler. Both are addressed by name.
 - **Limiting SEO scanners.** Link-analysis bots crawl aggressively and are frequently blocked outright.

```
`User-agent: * Disallow: /cart/ Disallow: /search Disallow: /*?utm_source= User-agent: Yandex Disallow: /cart/ Disallow: /search Disallow: /*?utm_source= Clean-param: sort&view&utm_source /catalog/ User-agent: Google-Extended Disallow: / User-agent: AhrefsBot Disallow: / Sitemap: https://example.com/sitemap.xml`
```

 Note that the `Yandex` group repeats all three shared rules. Without that repetition Yandex would end up with an empty rule set. Also note that `Yandex` covers all of that engine's bots, while Google's base name is `Googlebot`. Verify exact bot names in the official documentation rather than copying from examples — the internet holds more obsolete names than current ones.

## robots.txt and Security: A Blocked Directory Becomes a Public Hint

 robots.txt is readable by anyone and is the first thing any scanner requests. Lines like

```
`Disallow: /backup-2026/ Disallow: /old-admin/ Disallow: /internal-reports/`
```

 work as a table of contents: "here are three places worth looking at". Search crawlers will stay away; automated vulnerability scanners will not.

> No exceptions: if a section must not be seen by outsiders, it needs authentication or a 403/404. Listing such a path in robots.txt does not protect it — it advertises it.

 - Restrict service areas at the server or application level — by session, by IP, by Basic Auth.
 - If a section really must be excluded from crawling, prefer a wildcard over an exact name: `Disallow: /*/reports/` reveals less than a full path.
 - Never list backups, dumps or archives. They should not be inside the web root in the first place.
 - Do not rely on an "unguessable" directory name — it leaks through logs, referrers and third-party crawlers anyway.

## robots.txt vs llms.txt: How They Differ and Coexist

 They are two different files with two different jobs, and neither replaces the other.

 `robots.txt``llms.txt` PurposeAllow or block crawling of URLsShow a model what matters on the site, in a convenient form Format`name: value` directivesMarkdown: heading, summary, curated link lists StatusRFC 9309, supported by every engineA community proposal; adoption is voluntary and partial NatureRestrictiveAdvisory — a shop window Location`/robots.txt`, host root only`/llms.txt`, also the root If absentCrawling is fully allowedNothing happens — models just read the site normally They coexist neatly: robots.txt decides *whether to let* a given AI bot in, llms.txt decides *what to show* the ones you let in. If you block a bot in robots.txt, having an llms.txt changes nothing — the file still has to be fetched, and fetching requires access.

```
`# Let AI crawlers reach content but not service areas User-agent: * Disallow: /cart/ Disallow: /personal/ Disallow: /search User-agent: GPTBot Disallow: /cart/ Disallow: /personal/ Allow: / Sitemap: https://example.com/sitemap.xml`
```

 A full breakdown of AI bot names and allow/block strategies is in [robots.txt and AI crawlers](/en/articles/robots-txt-ai-crawlers). How to build the file itself is covered in the [llms.txt guide](/en/articles/llms-txt-guide).

## How to Check robots.txt

### Checking from the command line

 Before opening any validator, confirm the file is served correctly at all. Three things: a 200 response, `text/plain`, and no HTML in the body.

```
`# Headers: status code and Content-Type curl -sSI https://example.com/robots.txt # Full contents curl -sS https://example.com/robots.txt # Check for redirects and where they land curl -sSIL -o /dev/null -w '%{http_code} %{content_type} %{url_effective}\n' \ https://example.com/robots.txt # Look for a BOM at the start of the file (should be empty) curl -sS https://example.com/robots.txt | head -c 3 | xxd # Is there a fatal site-wide block? curl -sS https://example.com/robots.txt | grep -nE '^\s*Disallow:\s*/\s*$' # The same file on a subdomain — it is separate curl -sSI https://shop.example.com/robots.txt`
```

 If `grep` finds `Disallow: /` on a production domain, stop there — that is an incident.

### enterno.io tools

 - [robots.txt checker](/en/robots-checker) — parses the file, lists groups and rules, highlights syntax errors and tests a specific URL: allowed or blocked, and by which rule.
 - [HTTP header checker](/en/http-headers) — status code, `Content-Type`, redirects and the `X-Robots-Tag` header. It also reveals whether robots.txt is being served as HTML.
 - [SEO audit](/en/seo-audit) — looks at robots.txt together with the rest of the technical stack: sitemap, canonicals, meta tags, page availability.
 - [llms.txt checker](/en/llms-txt) — if you are configuring AI crawler access, confirm the companion file is present and valid.

### Webmaster consoles

 - **Google Search Console.** The standalone robots.txt tester has been retired; the property settings now contain a robots.txt report showing which files were found, when they were fetched, with what status and whether any fetch failed. To test a single address, use the URL inspection tool — it reports whether the URL is blocked by robots.txt.
 - **Yandex Webmaster.** Its tools include a robots.txt analyser: edit the content in the form and run a list of URLs to get a verdict for each.
 - **Bing Webmaster Tools.** Ships its own robots.txt tester with an editor.

> Test specific URLs rather than "the file as a whole": the homepage, a listing page, a product page, a stylesheet and a script. A syntactically valid robots.txt that blocks CSS is valid and harmful at the same time.

### When you need X-Robots-Tag instead

 For files that cannot carry a meta tag — PDF, DOCX, images, JSON — indexing is controlled with an HTTP header.

```
`# nginx: deindex documents and an internal section location ~* \.(pdf|docx?|xlsx?)$ { add_header X-Robots-Tag "noindex" always; } location ^~ /internal/ { add_header X-Robots-Tag "noindex, nofollow" always; } # Apache equivalent # <FilesMatch "\.(pdf|docx?|xlsx?)$"> # Header set X-Robots-Tag "noindex" # </FilesMatch>`
```

 Important: for the crawler to see this header, the path **must not** be blocked in robots.txt. Applying `Disallow` and `X-Robots-Tag` to the same URL is pointless.

## Common robots.txt Mistakes: Symptom → Cause → Check → Fix

 SymptomCauseHow to checkFix The whole site vanished from search`Disallow: /` shipped from staging, or a CMS visibility toggle is on`curl -sS https://site/robots.txt`Remove the line, request a recrawl, audit CMS settings The first directive is ignoredA BOM at the start of the file`curl -sS … | head -c 3 | xxd`Re-save as UTF-8 without BOM A rule is ignored by one engine onlyThat bot has its own named group without the ruleList every `User-agent` in the fileRepeat the full rule set inside the named group An unrelated section got blocked`Disallow: /admin` without a trailing slash also matched `/administration`Test the specific URL in a validatorAdd the slash: `Disallow: /admin/` Results show URLs with no descriptionThe URL is blocked but indexed through links`site:` search, webmaster reportsUnblock, add `noindex`, wait for a recrawl Mobile usability flagged as poorCSS and JS are blockedTest a stylesheet and a script URL in a validatorAdd `Allow` rules for static assets The sitemap "cannot be found"Relative path in `Sitemap`, or the sitemap is disallowedOpen the sitemap URL, review the rulesUse an absolute URL, remove the block Crawling stopped with no file changes`/robots.txt` returned 5xx during an outageMonitor the file's status codeServe it statically, not from the application The file looks right but is not appliedServed as `text/html`, or not at the root`curl -sSI` plus a path checkMove it to the root, serve `text/plain` A subdomain is crawled entirelyThe subdomain has no robots.txt of its own`curl -sSI https://sub.site/robots.txt`Create a separate file on the subdomain

### The three most frequent mistakes

 **A path without a leading slash.** `Disallow: catalog/` does nothing: the value must start with a slash, otherwise the rule is ignored or handled unpredictably.

 **Relying on line order.** "I put Allow above Disallow, so Allow wins" — no. Rule length decides. The only reliable way to be sure is to run the specific URL through a validator.

 **Fighting duplicates with robots.txt alone.** A blocked duplicate stays in the index if anything links to it, and simultaneously loses the ability to pass a `canonical`. Canonicals come first; `Disallow` joins in when the sheer volume of junk URLs genuinely hurts crawling.

## Frequently Asked Questions

### Is a robots.txt file mandatory?

 Formally, no. With no file and a 404 on that path, crawlers assume there are no restrictions and crawl everything. It is still worth having one for the `Sitemap` line and explicit blocks on service areas — cheaper than cleaning junk out of the index later.

### How do I block a whole site with robots.txt?

 With `User-agent: *` and `Disallow: /`. But that blocks *crawling*, not *indexing*: already-indexed URLs stay in results without descriptions. To genuinely remove a site you need `noindex` on the pages while robots.txt keeps them crawlable — and staging environments belong behind authentication.

### What does Disallow mean in robots.txt?

 "Do not request URLs starting with this prefix." The value is matched as a path prefix, not as a folder name, and it includes the query string. An empty `Disallow:` means the opposite: no restrictions.

### How do I test robots.txt online?

 Open the [robots.txt checker](/en/robots-checker), enter the domain and a specific URL — it shows whether the address is allowed and which rule decided it. It is also worth reading the robots.txt report in Google Search Console and the analyser in Yandex Webmaster: they show what the engine actually received, not what sits on your disk.

### Does a subdomain need its own robots.txt?

 Yes. robots.txt applies to a scheme + host + port combination. The file on `example.com` does not govern `shop.example.com`; the subdomain needs its own file at its own root.

### Why is a page blocked in robots.txt still in search results?

 Because robots.txt prevents fetching, not indexing. If links point at the URL, an engine can list it without a snippet. Only `noindex` removes it — and only if the crawler is allowed to load the page.

### Does Crawl-delay affect crawl rate?

 Google does not support it. Yandex stopped honouring it and moved crawl rate into its webmaster panel. Bing does read it. If crawler load is a real problem, server-side caching and rate limiting are far more dependable.

### Can I block an images folder in robots.txt?

 Technically yes, but think twice: the site disappears from image search, and if layout images live in the same folder, the crawler cannot render pages. Keep media crawlable and block only what truly must stay out of the index.

## Checklist: A Correct robots.txt

 - The file sits at `/robots.txt` in the host root and returns 200.
 - `Content-Type: text/plain`, UTF-8 encoding, no BOM.
 - It is served statically and does not depend on the application being up.
 - There is no `Disallow: /` on the production domain.
 - Every path starts with a slash; directory paths also end with one.
 - No blank lines inside a group; blank lines only between groups.
 - Any named group (Yandex, Googlebot) repeats the full rule set.
 - CSS, JavaScript, fonts and layout images are crawlable.
 - At least one `Sitemap` line with an absolute HTTPS URL, and the sitemap path is not blocked.
 - The file lists no backups, dumps or sensitive paths.
 - No URL carries both `Disallow` and `noindex`.
 - Every subdomain has its own robots.txt.
 - The file has been tested against real URLs: homepage, listing, product page, CSS, JS.
 - The contents are compared against a known-good baseline after every release.

 The wider set of technical checks that robots.txt is one part of is collected in the [SEO audit checklist](/en/articles/seo-audit-checklist).

---

Source: https://enterno.io/en/articles/robots-txt-guide
