robots.txt Guide: Syntax, Rules, Testing and Ready-Made Files

In short: robots.txt is a plain text file at the root of a host that tells crawlers which URLs they may fetch. It controls crawling, not indexing: a page blocked in robots.txt can still appear in search results — without a description. The syntax is defined in RFC 9309: User-agent, Disallow, Allow, plus the Sitemap extension.

What robots.txt Is and How a Crawler Reads It

robots.txt is a plain text file that must live at exactly one address: https://example.com/robots.txt. No other filename, no subdirectory. A crawler requests that path and looks nowhere else. Inside it is a list of name: value lines separated by line breaks.

The sequence looks like this:

1. Before crawling a host, the bot issues GET /robots.txt against the same scheme, host and port.
2. The file is parsed and cached — typically for around a day, longer if caching headers say so.
3. For every URL the crawler picks one matching group of rules and evaluates the path against it.
4. If the path is disallowed, the crawler never requests the page. It knows the URL exists; it just never sees the content.

The protocol is voluntary by design. robots.txt is not a firewall, not an access rule and not authentication. Major engines (Google, Bing, Yandex) obey it because it is in their interest. Scrapers, content harvesters and vulnerability scanners do not. So robots.txt solves exactly one problem: it saves crawl budget and keeps junk URLs out of the fetch queue.

robots.txt is a request, not a restriction. Anything that genuinely must not be public belongs behind authentication or behind a 403/404. The file itself stays publicly readable by anyone with a browser.

Why restrict crawling at all? Because a crawler has a budget of requests per site per unit of time. If 200,000 sorting and filtering URLs consume that budget, product pages get visited rarely. Removing junk redistributes the budget toward pages that actually need to rank.

robots.txt Syntax: Standard Directives and Everything That Is Not

For decades robots.txt was a 1994 convention with no formal specification, so parsers disagreed on details. The protocol is now standardised in RFC 9309, the Robots Exclusion Protocol. It defines exactly three things: the User-agent line and the Allow and Disallow rules. Everything else is an extension with varying support.

Parsing rules

• One directive per line: name, colon, value.
• Directive names are case-insensitive. Disallow, disallow and DISALLOW are the same. So are user-agent names.
• Path values are case-sensitive. /Admin/ and /admin/ are different paths. This is the most common reason a rule "does nothing".
• # starts a comment that runs to the end of the line.
• The file must be UTF-8. Characters outside the allowed set may be discarded.
• An empty Disallow: value means "nothing is disallowed". It is an allow rule, not a mistake.

A minimal valid file

# Minimal working robots.txt
User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

This allows the whole site and points at the sitemap. Strictly speaking a site with no restrictions needs no robots.txt at all, but returning a 404 there is not ideal: some audit tools flag it, and the Sitemap line is convenient to keep in this exact place.

What is not part of the standard

You will meet the directives below in other people's files and in old tutorials. None of them is in RFC 9309, and support varies widely.

• Crawl-delay — a request to pause between fetches. Google has never supported it and silently ignores it. Yandex dropped it in favour of a crawl-rate setting in its webmaster panel. Bing honours it. As a load-control mechanism it is unreliable — fix the cause on the server instead.
• Host — historically told Yandex which mirror was canonical. Yandex has retired it; the canonical host is now determined by 301 redirects and webmaster settings. Leaving it in the file does no damage but confuses whoever reads it next.
• Clean-param — a Yandex extension that lists query parameters which do not change page content, so URLs can be consolidated. Yandex only; Google does not parse it.
• Noindex: inside robots.txt — does nothing. No such directive exists in the protocol and no engine executes it. Use a meta tag or an HTTP header instead.
• Request-rate, Visit-time — relics nobody honours today.

Directive reference

Directive	Support	What it does	Typical mistake
User-agent	Standard, all crawlers	Opens a group and names the crawler it applies to	A typo in the bot name — the group applies to nobody and the bot falls back to *
Disallow	Standard, all crawlers	Blocks URLs starting with the given prefix	Missing leading slash; /admin instead of /admin/, which also blocks /administration
Allow	Standard (Google, Bing, Yandex)	Explicitly permits a prefix inside a disallowed area	Relying on line order: precedence comes from rule length, not position
Sitemap	Extension, widely supported	Points to an absolute sitemap URL	A relative path such as Sitemap: /sitemap.xml — the line is ignored
Crawl-delay	Bing; not Google; retired by Yandex	Asks for a delay between requests	Used as a substitute for proper caching and rate limiting
Host	Obsolete, retired by Yandex	Used to declare the canonical mirror	The file looks configured while the line does nothing
Clean-param	Yandex only	Consolidates URLs differing by insignificant parameters	Expecting Google to read it
Noindex	Not supported anywhere	Nothing	The page stays indexed while the owner believes otherwise

The Group Rule: How Several User-agent Lines Share One Block

A robots.txt file is not a flat list — it is a set of groups. A group consists of one or more consecutive User-agent lines followed by Allow/Disallow rules. The first rule line closes the list of names: the next User-agent line starts a new group.

# One group for three crawlers: shared rules
User-agent: Googlebot
User-agent: Yandex
User-agent: bingbot
Disallow: /cart/
Disallow: /search

# A separate group for everybody else
User-agent: *
Disallow: /cart/
Disallow: /search
Disallow: /api/

Googlebot, Yandex and bingbot get two rules; everyone else gets three. Note that the * group does not apply at all to the three named bots. That is the main trap.

A crawler picks exactly one group

The bot looks for the group whose name matches it most specifically. Once found, it obeys that group only and ignores the * section entirely. The classic failure:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search
Disallow: /*?utm_source=

User-agent: Yandex
Clean-param: sort&view /catalog/

The author meant to add "one setting for Yandex". What actually happened is that all four restrictions were lifted for Yandex: its bot found its own group, and that group contains no Disallow at all. The fix is to repeat the shared rules inside the named group in full.

The rule is simple: if you create a named section for a crawler, that section must contain the complete rule set for it. Groups are never inherited and never merged with *.

Duplicate groups and blank lines

If the same User-agent appears twice, modern parsers — Google's among them — merge those groups into one. Do not rely on it, though: audit tools and older parsers behave differently.

Blank lines are a separate story. The original 1994 convention used a blank line as the record separator, and some parsers still treat it as the end of a group. Google's current parser ignores blank lines. The practical conclusion: never put a blank line inside a group. Keep groups monolithic and use blank lines only between groups — then every parser reads the file the same way.

# Bad: some parsers end the group at the blank line
User-agent: *
Disallow: /admin/

Disallow: /cart/

# Good: monolithic group, blank line only between groups
User-agent: *
Disallow: /admin/
Disallow: /cart/

User-agent: AhrefsBot
Disallow: /

How Paths Are Matched: Wildcards and Allow Precedence

Prefix matching

A Disallow value is compared against the URL path as a prefix — not as a full path and not as a directory name. Disallow: /admin blocks /admin, /admin/, /admin/users and also /administrator and /admin-guide.html. If you mean a directory, add the trailing slash: Disallow: /admin/.

The query string is part of the match. Disallow: /*?sort= catches /catalog/?sort=price. Scheme and host take no part in matching.

The * and $ wildcards

• * matches any sequence of characters, including none. Disallow: /*/print catches /catalog/print and /blog/2026/print.
• $ anchors the end of the URL. Disallow: /*.pdf$ blocks /docs/manual.pdf but leaves /docs/manual.pdf.html and /file.pdf?v=2 crawlable, because characters follow .pdf.
• ?, . and & are ordinary characters here — there is no regular-expression behaviour.
• Google, Bing and Yandex all support both wildcards.

Rule	URL	Result
Disallow: /admin	/administrator/login	Blocked — the prefix matches
Disallow: /admin/	/administrator/login	Allowed
Disallow: /*.pdf$	/docs/a.pdf?v=2	Allowed — $ requires end of string
Disallow: /*?	/catalog/?page=2	Blocked — every URL with a query string
Disallow: catalog/	/catalog/	Does not work — no leading slash
Disallow:	any	Allowed — an empty value blocks nothing

Precedence: the longest rule wins, not the first

The most widespread misconception is "whatever comes last wins" or "Allow always beats Disallow". The actual rule: the most specific rule applies — the one with the longest path value. Line order is irrelevant. If two rules are the same length, the less restrictive one wins, which means Allow.

User-agent: *
Disallow: /catalog/
Allow: /catalog/deals/

For /catalog/deals/lamp both rules match. /catalog/deals/ is 15 characters, /catalog/ is 9. Allow wins and the page is crawled. Swap the two lines and nothing changes.

# The reverse case: Allow is shorter, so Disallow wins
User-agent: *
Allow: /catalog/
Disallow: /catalog/private/

# /catalog/private/doc is blocked (18 > 9)

* and $ count as ordinary characters when measuring length, so rules with wildcards sometimes win unexpectedly. When in doubt, test the specific URL in a validator instead of counting characters in your head.

The Big Misconception: robots.txt Does Not Remove a Page From the Index

This is the point that damages real sites most often. robots.txt blocks crawling, not indexing. The difference matters.

If a blocked page has inbound links, internal or external, a search engine may index the URL anyway. It never saw the content, so the result shows a bare URL or a title assembled from anchor text, plus a note along the lines of "No information is available for this page". The page is in the results. You did not remove it.

The classic trap: a page is blocked in robots.txt and carries <meta name="robots" content="noindex">. The crawler cannot fetch the page, so it cannot read the noindex, so it never acts on it. The two "blocks" cancel each other out and the URL stays in the results for years.

The correct removal sequence

1. Unblock the page in robots.txt — remove the matching Disallow.
2. Add <meta name="robots" content="noindex, follow"> or the X-Robots-Tag: noindex header.
3. Wait for a recrawl. This takes weeks; you can nudge it from the webmaster consoles.
4. Only after the URL has left the index, block the path in robots.txt if you want to stop spending crawl budget on it.

For more on the meta tags involved, see the guide to title, description and meta tags. If the opposite is happening and pages will not appear at all, see why a site is not in search results.

Which tool solves which job

Goal	Right tool	Why not robots.txt
Keep crawlers out of the admin area	robots.txt plus authentication	robots.txt saves budget; only auth restricts access
Remove a page from the index	noindex meta tag or X-Robots-Tag	A blocked page is never fetched, so noindex is never read
Keep a staging domain out of search	HTTP Basic Auth (401)	A staging robots.txt easily ships to production and kills the live site
Deal with filter and sort duplicates	canonical, plus Disallow on parameters at scale	A blocked duplicate cannot pass its canonical signal — the bot never sees it
Keep PDFs and documents out of the index	X-Robots-Tag: noindex header	You cannot put a meta tag in a binary file, and robots.txt does not deindex
Hide confidential data	Authentication, 403/404, move it out of the web root	robots.txt is public and reads like a table of contents
Allow or block AI crawlers	robots.txt by bot name; llms.txt for content curation	Two different jobs: access versus presentation
Reduce crawler load	Server-side rate limiting, caching, webmaster crawl-rate settings	Google ignores Crawl-delay and Yandex retired it

How to Block a Site With robots.txt — and When That Backfires

Blocking an entire site looks like this:

# Block the whole site for every crawler
User-agent: *
Disallow: /

Note the difference that breaks sites:

• Disallow: / — the entire site is blocked.
• Disallow: — nothing is blocked.

One character. A typo here is the most expensive typo in SEO.

When Disallow: / is genuinely appropriate

• A technical domain, a mirror or a staging subdomain that should never rank.
• A site under construction that has never been indexed.
• A parked domain bought for a redirect.

Why it is a poor way to hide a site

First, if the site is already indexed, Disallow: / will not remove it — see the previous section. Worse, you cut off the crawler's ability to ever read a noindex, so those URLs stay in the results as bare links.

Second, a staging file with Disallow: / is a time bomb. It lives in the repository, somebody deploys the wrong branch, and the production site goes dark. It is usually discovered two or three weeks later, once traffic has already dropped.

Third, a blocked staging robots.txt is public and effectively announces "there is a site here and somebody is hiding it". Scanners find that interesting.

Close a staging environment with authentication, not robots.txt. Return 401 to everyone but your team: no crawler can obtain the content at all, and robots.txt stops being a single point of failure on deploy.

# nginx: lock a staging domain properly
server {
    server_name staging.example.com;

    auth_basic           "Staging";
    auth_basic_user_file /etc/nginx/.htpasswd;

    # belt and braces in case auth is ever removed
    add_header X-Robots-Tag "noindex, nofollow" always;
}

After every release it pays to verify that the production robots.txt has not been replaced. The cheapest protection is content monitoring: if Disallow: / ever appears, you find out that day rather than a month later.

Sitemap in robots.txt: Declaring Your Sitemap Correctly

The Sitemap directive is an extension described at sitemaps.org and supported by Google, Bing and Yandex. Its rules differ from the other directives.

• Absolute URLs only. Sitemap: /sitemap.xml is a dead line and is skipped. Write Sitemap: https://example.com/sitemap.xml.
• Multiple lines are allowed. Declare as many sitemaps as you have: products, articles, images.
• The directive is not part of any group. It is global, regardless of which User-agent blocks surround it. For readability, keep it in its own block at the top or bottom of the file.
• Scheme and host must match production. If the site is on HTTPS and the sitemap is declared over http://, you are sending the crawler through a needless redirect.
• Never block the sitemap path. Combining Disallow: /*.xml$ with a Sitemap line is a reliable way to break your own indexing.

User-agent: *
Disallow: /cart/
Disallow: /search
Disallow: /*?utm_source=

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-products.xml
Sitemap: https://example.com/sitemap-images.xml

With many sitemaps, declare a single sitemap index and list the rest inside it — robots.txt stays short. Sitemap index structure, priorities and lastmod are covered in the sitemap.xml guide.

Where robots.txt Must Live and What the Server Must Return

One host, one file

robots.txt applies to a combination of scheme + host + port. Several non-obvious consequences follow:

• https://example.com/robots.txt and https://shop.example.com/robots.txt are different files. A subdomain needs its own; the root file does not cover it.
• https://example.com and http://example.com are formally different origins too. In practice a site serves the same file over both, which is what you want.
• https://example.com:8443/ is again a separate file.
• A file in a subfolder does nothing: /blog/robots.txt is never requested.
• Rules do not reach other hosts. If your assets sit on a CDN domain, they are governed by that domain's robots.txt.

Status codes and how crawlers read them

Response to /robots.txt	Crawler behaviour	Risk
200 with text	The file is parsed and applied	Normal
404 or another 4xx	Treated as "no restrictions": everything is allowed	The whole site is crawled, service sections included
401, 403	Treated as an absent file — usually everything allowed	Common when the entire site sits behind auth
5xx	Treated as unreachable — the crawler assumes everything is disallowed	Crawling stops completely for the duration of the outage
3xx	Redirects are followed, but only a limited number (around five)	A redirect chain or loop means the file is never retrieved
Timeout, reset	Same as 5xx: crawling is paused	An unstable server slows indexing

The most underestimated incident: the site goes down and /robots.txt starts returning 500 along with everything else. To a search engine that reads as "the whole site is closed". If the 5xx persists, crawling halts entirely. This is why robots.txt should be served as a static file rather than rendered by the application.

A full breakdown of status codes and their SEO consequences lives in the HTTP status codes reference.

Size, encoding, Content-Type

• Size. RFC 9309 requires parsers to handle at least 500 KiB. Anything beyond that may be dropped. A 2,000-line file almost always means parameters should be handled with canonicals rather than enumerated.
• Encoding. UTF-8 only. Non-ASCII characters in comments are fine, but paths are safer percent-encoded, exactly as they appear in requests.
• BOM. Save the file without a BOM. Google ignores a leading BOM, but not every parser does: three invisible bytes attach to the first line and User-agent: * stops being recognised. The symptom is distinctive — "the first directive is ignored, the rest work".
• Content-Type. The file must be served as text/plain. If a framework serves it as text/html — typical when a controller renders robots.txt — some crawlers will not process it.
• Line endings. LF and CRLF are both fine. HTML inside the file is not: no <pre>, no page wrapper.

Ready-Made robots.txt for WordPress, Tilda and Bitrix

The principle is the same for any CMS: block service areas, internal search, the cart and parameterised URLs — and do not block CSS, JavaScript or images. Without them the crawler cannot render the page or judge the mobile layout.

Blocking /wp-includes/, /bitrix/ or /assets/ wholesale is not "security", it is broken rendering. The engine sees an unstyled page and evaluates it accordingly.

robots.txt for WordPress

WordPress has a quirk: with no physical file in the root, it serves a virtual robots.txt that exists only in the response. The default content blocks /wp-admin/ while allowing admin-ajax.php.

Two practical consequences. First, any physical robots.txt in the root fully overrides the virtual one. Second, and far nastier: the Settings → Reading → Discourage search engines from indexing this site checkbox replaces the virtual file with User-agent: * and Disallow: /. It gets ticked during development and forgotten. If a site suddenly disappears from search, check that box first.

# robots.txt for WordPress
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Disallow: /?s=
Disallow: /search/
Disallow: /*?replytocom=
Disallow: /*?utm_source=
Disallow: /*?utm_medium=
Disallow: /*?utm_campaign=
Disallow: /*?fbclid=
Disallow: /*?gclid=

# Keep CSS, JS and media crawlable
Allow: /wp-content/uploads/
Allow: /wp-includes/js/
Allow: /*.css$
Allow: /*.js$

Sitemap: https://example.com/wp-sitemap.xml

What is deliberately absent: blanket blocks on /wp-content/plugins/ and /wp-includes/. Plugins ship front-end CSS and JS, and blocking them breaks rendering. Tags, archives and pagination are better handled with meta tags than with robots.txt — otherwise the crawler sees neither noindex nor canonical.

robots.txt for a Tilda site

Tilda generates robots.txt on its side and serves it automatically once the site is published to your own domain. The content can be overridden in the site settings, where the platform provides a field for your own version.

• Tilda styles and scripts load from its own static domains. Your robots.txt has no effect on them — and you neither need nor can block them.
• A site published to the platform's service subdomain is a separate host with its own robots.txt that you do not control. Keep the preview out of search by not linking to it and not verifying it in webmaster consoles.
• The main things worth adding by hand are the Sitemap line and blocks for utility pages such as thank-you screens.

# robots.txt for a Tilda site
User-agent: *
Disallow: /thanks
Disallow: /*?utm_source=
Disallow: /*?utm_medium=
Disallow: /*?utm_campaign=

Sitemap: https://example.com/sitemap.xml

robots.txt for Bitrix

Bitrix generates the file from its SEO module, and the default output blocks /bitrix/ entirely. That is exactly where manual editing is needed: /bitrix/ holds js, templates and compiled CSS caches — precisely what the crawler needs in order to render.

# robots.txt for Bitrix
User-agent: *
Disallow: /bitrix/
Disallow: /local/
Disallow: /auth/
Disallow: /personal/
Disallow: /cart/
Disallow: /search/
Disallow: /upload/
Disallow: /*PAGEN_
Disallow: /*?print=
Disallow: /*?set_filter=
Disallow: /*?clear_cache=
Disallow: /*?utm_source=

# Give the crawler the layout and scripts back
Allow: /bitrix/js/
Allow: /bitrix/templates/
Allow: /bitrix/cache/css/
Allow: /bitrix/cache/js/
Allow: /local/templates/
Allow: /upload/iblock/
Allow: /*.css$
Allow: /*.js$

Sitemap: https://example.com/sitemap.xml

About /upload/: never block it wholesale if it stores product photography — the site then vanishes from image search. That is why /upload/iblock/ is allowed back in the example above.

Faceted filters are the classic Bitrix problem: they generate a combinatorial explosion of URLs. Blocking them via Disallow: /*?set_filter= is reasonable, but filter pages must also carry a correct canonical, and genuinely valuable filter combinations are better promoted to their own clean-URL landing pages and left crawlable.

Separate Sections for Google and Yandex: When You Actually Need Them

By default you do not. A single User-agent: * group covers every engine, and the shorter the file, the fewer mistakes it contains. Named sections are justified in four cases.

1. You need Clean-param. Only Yandex parses it, so it can physically live only in a Yandex group.
2. The rules genuinely differ. For example, one engine gets a set of images another does not, or a section is open to one and closed to the other.
3. AI crawler control. Google separates its model-training agent from its search crawler (Google-Extended), and Yandex likewise runs a separate additional crawler. Both are addressed by name.
4. Limiting SEO scanners. Link-analysis bots crawl aggressively and are frequently blocked outright.

User-agent: *
Disallow: /cart/
Disallow: /search
Disallow: /*?utm_source=

User-agent: Yandex
Disallow: /cart/
Disallow: /search
Disallow: /*?utm_source=
Clean-param: sort&view&utm_source /catalog/

User-agent: Google-Extended
Disallow: /

User-agent: AhrefsBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Note that the Yandex group repeats all three shared rules. Without that repetition Yandex would end up with an empty rule set. Also note that Yandex covers all of that engine's bots, while Google's base name is Googlebot. Verify exact bot names in the official documentation rather than copying from examples — the internet holds more obsolete names than current ones.

robots.txt and Security: A Blocked Directory Becomes a Public Hint

robots.txt is readable by anyone and is the first thing any scanner requests. Lines like

Disallow: /backup-2026/
Disallow: /old-admin/
Disallow: /internal-reports/

work as a table of contents: "here are three places worth looking at". Search crawlers will stay away; automated vulnerability scanners will not.

No exceptions: if a section must not be seen by outsiders, it needs authentication or a 403/404. Listing such a path in robots.txt does not protect it — it advertises it.

• Restrict service areas at the server or application level — by session, by IP, by Basic Auth.
• If a section really must be excluded from crawling, prefer a wildcard over an exact name: Disallow: /*/reports/ reveals less than a full path.
• Never list backups, dumps or archives. They should not be inside the web root in the first place.
• Do not rely on an "unguessable" directory name — it leaks through logs, referrers and third-party crawlers anyway.

robots.txt vs llms.txt: How They Differ and Coexist

They are two different files with two different jobs, and neither replaces the other.

	robots.txt	llms.txt
Purpose	Allow or block crawling of URLs	Show a model what matters on the site, in a convenient form
Format	name: value directives	Markdown: heading, summary, curated link lists
Status	RFC 9309, supported by every engine	A community proposal; adoption is voluntary and partial
Nature	Restrictive	Advisory — a shop window
Location	/robots.txt, host root only	/llms.txt, also the root
If absent	Crawling is fully allowed	Nothing happens — models just read the site normally

They coexist neatly: robots.txt decides whether to let a given AI bot in, llms.txt decides what to show the ones you let in. If you block a bot in robots.txt, having an llms.txt changes nothing — the file still has to be fetched, and fetching requires access.

# Let AI crawlers reach content but not service areas
User-agent: *
Disallow: /cart/
Disallow: /personal/
Disallow: /search

User-agent: GPTBot
Disallow: /cart/
Disallow: /personal/
Allow: /

Sitemap: https://example.com/sitemap.xml

A full breakdown of AI bot names and allow/block strategies is in robots.txt and AI crawlers. How to build the file itself is covered in the llms.txt guide.

How to Check robots.txt

Checking from the command line

Before opening any validator, confirm the file is served correctly at all. Three things: a 200 response, text/plain, and no HTML in the body.

# Headers: status code and Content-Type
curl -sSI https://example.com/robots.txt

# Full contents
curl -sS https://example.com/robots.txt

# Check for redirects and where they land
curl -sSIL -o /dev/null -w '%{http_code} %{content_type} %{url_effective}\n' \
  https://example.com/robots.txt

# Look for a BOM at the start of the file (should be empty)
curl -sS https://example.com/robots.txt | head -c 3 | xxd

# Is there a fatal site-wide block?
curl -sS https://example.com/robots.txt | grep -nE '^\s*Disallow:\s*/\s*$'

# The same file on a subdomain — it is separate
curl -sSI https://shop.example.com/robots.txt

If grep finds Disallow: / on a production domain, stop there — that is an incident.

enterno.io tools

• robots.txt checker — parses the file, lists groups and rules, highlights syntax errors and tests a specific URL: allowed or blocked, and by which rule.
• HTTP header checker — status code, Content-Type, redirects and the X-Robots-Tag header. It also reveals whether robots.txt is being served as HTML.
• SEO audit — looks at robots.txt together with the rest of the technical stack: sitemap, canonicals, meta tags, page availability.
• llms.txt checker — if you are configuring AI crawler access, confirm the companion file is present and valid.

Webmaster consoles

• Google Search Console. The standalone robots.txt tester has been retired; the property settings now contain a robots.txt report showing which files were found, when they were fetched, with what status and whether any fetch failed. To test a single address, use the URL inspection tool — it reports whether the URL is blocked by robots.txt.
• Yandex Webmaster. Its tools include a robots.txt analyser: edit the content in the form and run a list of URLs to get a verdict for each.
• Bing Webmaster Tools. Ships its own robots.txt tester with an editor.

Test specific URLs rather than "the file as a whole": the homepage, a listing page, a product page, a stylesheet and a script. A syntactically valid robots.txt that blocks CSS is valid and harmful at the same time.

When you need X-Robots-Tag instead

For files that cannot carry a meta tag — PDF, DOCX, images, JSON — indexing is controlled with an HTTP header.

# nginx: deindex documents and an internal section
location ~* \.(pdf|docx?|xlsx?)$ {
    add_header X-Robots-Tag "noindex" always;
}

location ^~ /internal/ {
    add_header X-Robots-Tag "noindex, nofollow" always;
}

# Apache equivalent
# <FilesMatch "\.(pdf|docx?|xlsx?)$">
#   Header set X-Robots-Tag "noindex"
# </FilesMatch>

Important: for the crawler to see this header, the path must not be blocked in robots.txt. Applying Disallow and X-Robots-Tag to the same URL is pointless.

Common robots.txt Mistakes: Symptom → Cause → Check → Fix

Symptom	Cause	How to check	Fix
The whole site vanished from search	Disallow: / shipped from staging, or a CMS visibility toggle is on	curl -sS https://site/robots.txt	Remove the line, request a recrawl, audit CMS settings
The first directive is ignored	A BOM at the start of the file	curl -sS … | head -c 3 | xxd	Re-save as UTF-8 without BOM
A rule is ignored by one engine only	That bot has its own named group without the rule	List every User-agent in the file	Repeat the full rule set inside the named group
An unrelated section got blocked	Disallow: /admin without a trailing slash also matched /administration	Test the specific URL in a validator	Add the slash: Disallow: /admin/
Results show URLs with no description	The URL is blocked but indexed through links	site: search, webmaster reports	Unblock, add noindex, wait for a recrawl
Mobile usability flagged as poor	CSS and JS are blocked	Test a stylesheet and a script URL in a validator	Add Allow rules for static assets
The sitemap "cannot be found"	Relative path in Sitemap, or the sitemap is disallowed	Open the sitemap URL, review the rules	Use an absolute URL, remove the block
Crawling stopped with no file changes	/robots.txt returned 5xx during an outage	Monitor the file's status code	Serve it statically, not from the application
The file looks right but is not applied	Served as text/html, or not at the root	curl -sSI plus a path check	Move it to the root, serve text/plain
A subdomain is crawled entirely	The subdomain has no robots.txt of its own	curl -sSI https://sub.site/robots.txt	Create a separate file on the subdomain

The three most frequent mistakes

A path without a leading slash. Disallow: catalog/ does nothing: the value must start with a slash, otherwise the rule is ignored or handled unpredictably.

Relying on line order. "I put Allow above Disallow, so Allow wins" — no. Rule length decides. The only reliable way to be sure is to run the specific URL through a validator.

Fighting duplicates with robots.txt alone. A blocked duplicate stays in the index if anything links to it, and simultaneously loses the ability to pass a canonical. Canonicals come first; Disallow joins in when the sheer volume of junk URLs genuinely hurts crawling.

Frequently Asked Questions

Is a robots.txt file mandatory?

Formally, no. With no file and a 404 on that path, crawlers assume there are no restrictions and crawl everything. It is still worth having one for the Sitemap line and explicit blocks on service areas — cheaper than cleaning junk out of the index later.

How do I block a whole site with robots.txt?

With User-agent: * and Disallow: /. But that blocks crawling, not indexing: already-indexed URLs stay in results without descriptions. To genuinely remove a site you need noindex on the pages while robots.txt keeps them crawlable — and staging environments belong behind authentication.

What does Disallow mean in robots.txt?

"Do not request URLs starting with this prefix." The value is matched as a path prefix, not as a folder name, and it includes the query string. An empty Disallow: means the opposite: no restrictions.

How do I test robots.txt online?

Open the robots.txt checker, enter the domain and a specific URL — it shows whether the address is allowed and which rule decided it. It is also worth reading the robots.txt report in Google Search Console and the analyser in Yandex Webmaster: they show what the engine actually received, not what sits on your disk.

Does a subdomain need its own robots.txt?

Yes. robots.txt applies to a scheme + host + port combination. The file on example.com does not govern shop.example.com; the subdomain needs its own file at its own root.

Why is a page blocked in robots.txt still in search results?

Because robots.txt prevents fetching, not indexing. If links point at the URL, an engine can list it without a snippet. Only noindex removes it — and only if the crawler is allowed to load the page.

Does Crawl-delay affect crawl rate?

Google does not support it. Yandex stopped honouring it and moved crawl rate into its webmaster panel. Bing does read it. If crawler load is a real problem, server-side caching and rate limiting are far more dependable.

Can I block an images folder in robots.txt?

Technically yes, but think twice: the site disappears from image search, and if layout images live in the same folder, the crawler cannot render pages. Keep media crawlable and block only what truly must stay out of the index.

Checklist: A Correct robots.txt

• The file sits at /robots.txt in the host root and returns 200.
• Content-Type: text/plain, UTF-8 encoding, no BOM.
• It is served statically and does not depend on the application being up.
• There is no Disallow: / on the production domain.
• Every path starts with a slash; directory paths also end with one.
• No blank lines inside a group; blank lines only between groups.
• Any named group (Yandex, Googlebot) repeats the full rule set.
• CSS, JavaScript, fonts and layout images are crawlable.
• At least one Sitemap line with an absolute HTTPS URL, and the sitemap path is not blocked.
• The file lists no backups, dumps or sensitive paths.
• No URL carries both Disallow and noindex.
• Every subdomain has its own robots.txt.
• The file has been tested against real URLs: homepage, listing, product page, CSS, JS.
• The contents are compared against a known-good baseline after every release.

The wider set of technical checks that robots.txt is one part of is collected in the SEO audit checklist.