In short. A sitemap.xml is a machine-readable list of your site's canonical URLs in the sitemaps.org protocol format. It tells crawlers what to fetch, but it does not guarantee indexing and it is not a ranking factor. One file holds up to 50,000 URLs and 50 MB uncompressed; beyond that you need a sitemap index. Search engines effectively ignore changefreq and priority, and lastmod counts only when the dates are trustworthy.
What a sitemap.xml is and who actually needs one
An XML sitemap is a plain UTF-8 text file that lists page addresses in a fixed schema defined by the sitemaps.org protocol, version 0.9. The format has not changed since 2005 and is parsed identically by Google, Bing, Yandex and every other mainstream crawler — one of the few genuinely universal standards in SEO.
A crawler's default discovery mechanism is link following: take a known URL, fetch the HTML, extract every href, queue what you found. A sitemap is a second, independent discovery channel — a list of addresses that does not require a link path to exist at all.
That determines who gets real value from one:
- Large catalogues and stores. Tens or hundreds of thousands of URLs, deep nesting, product pages reachable only through filters and pagination — a crawler will not physically reach everything in a reasonable number of passes.
- New sites with no backlinks. There is nothing to discover: no external signals, and internal linking is still thin.
- Sites with weak internal structure. Sections no menu or listing links to — orphan pages, in the language of a technical audit.
- News and media projects. Content goes stale within hours, so discovery speed matters more than anything else.
- Multilingual projects. hreflang annotations are easier to maintain in a sitemap than in the
<head>of every page.
And who gets almost nothing: a 30-to-80-page site where everything is linked from a persistent menu and breadcrumbs. A crawler covers that in a single visit. The file will not hurt, but it will not fix anything either — if pages are missing from the index, the cause lies elsewhere, and it should be diagnosed the way described in why your site is not in search.
What a sitemap does — and what it does not
This is where most of the confusion lives, so let us be blunt about it.
What it actually does
- Hands the crawler a list of addresses. One request, thousands of URLs, instead of walking a link graph.
- Communicates a modification date — if you populate it honestly. That lets a crawler prioritise re-fetching what changed.
- Speeds up discovery of new and deeply buried pages. Discovery is not indexing, but without it indexing cannot happen at all.
- Gives you diagnostics. Webmaster consoles show "submitted" against "indexed" per file. It is the simplest way to measure what share of your content actually made it into the index.
What it does not do
- It does not guarantee indexing. A URL in the file is an invitation, not an obligation. The crawler still decides whether to fetch it and whether it deserves index space.
- It does not affect rankings. No tag in the protocol is a ranking factor.
priorityis a crawl-ordering hint within your own site, not page "weight" in results. - It does not override robots.txt or noindex. If a URL is blocked in robots.txt or marked
noindex, listing it in a sitemap changes nothing — it only creates a signal conflict. - It does not replace internal linking. Link structure passes authority and expresses hierarchy; a sitemap does neither.
- It does not fix duplicates. Canonical selection is the job of
rel="canonical"and host configuration, not the sitemap. - It does not trigger re-indexing on demand. Submitting the file does not launch an immediate crawl of every URL in it.
Rule. A sitemap is a crawl hint. If a page is thin, duplicated or technically unreachable, being listed will not improve its fate. The opposite is true: dumping junk into the file dilutes the signal and destroys your own diagnostics — you lose the ability to tell a content problem from your own noise.
Sitemap structure: urlset, url, loc, lastmod, changefreq, priority
A minimal valid file looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/</loc>
<lastmod>2026-07-14</lastmod>
</url>
<url>
<loc>https://example.com/catalog/windows/</loc>
<lastmod>2026-06-28T11:20:00+03:00</lastmod>
</url>
<url>
<loc>https://example.com/about/</loc>
</url>
</urlset>Four things matter here. First, the XML declaration must be on the very first line, with no whitespace or blank lines before it. Second, the xmlns attribute on the root element is a namespace identifier, not a URL that gets fetched — rewriting it to https:// breaks recognition of the file as a sitemap. Third, exactly one <loc> per <url>. Fourth, optional tags can simply be omitted, as in the third entry above.
How the tags are really treated
| Tag | Required | How it is really treated | Typical mistake |
|---|---|---|---|
<urlset> | Yes | Root element. Without the correct namespace the document is not recognised as a sitemap at all | Namespace dropped, rewritten to https, or replaced with a "similar" one — the file is parsed as arbitrary XML |
<url> | Yes | Container for a single entry. Exactly one <loc> inside | Two <loc> elements in one entry, or an entry with no <loc> |
<loc> | Yes | The only element every search engine reads without exception. An absolute URL on the same scheme and host as the sitemap file itself | http addresses on an https site, relative paths, unescaped &, non-ASCII characters without percent-encoding |
<lastmod> | No | Used only when the dates can be trusted. Google states plainly that the value is used when it is consistently and verifiably accurate. Inside an index file it is the most useful tag of all: it tells the crawler which child file is worth re-reading | The file build time is written instead of the content change date — so the whole site "updates" every night |
<changefreq> | No | Google ignores it entirely. Other crawlers treat it, at best, as a very weak hint. Real crawl frequency is driven by observed site behaviour, not by a declaration | daily on every page "so they visit more often" — the signal is discounted within a week |
<priority> | No | Google ignores it. The value is relative within one site and has never been a ranking factor | 1.0 on every URL, which is mathematically identical to having no priority at all |
<xhtml:link> | No | The hreflang extension. Read by Google; easier to maintain centrally than markup in <head> | No self-reference, and alternate sets that do not match between entries |
<sitemapindex> / <sitemap> | Index only | Root element and entry of an index file. Each entry holds a <loc> pointing at a regular sitemap, plus an optional <lastmod> | An index referencing another index, or mixing <url> and <sitemap> entries in one file |
lastmod in detail
The format is W3C Datetime: both the short 2026-07-14 and the full 2026-07-14T09:15:00+03:00 with a timezone are valid. Mixing both forms in one file is fine — parsers handle it.
The problem is not the format, it is truthfulness. A "significant change" means an edit to the main content, the structured data or the links on the page — not a footer year rollover, not a cache rebuild, not an automatic refresh of a "related products" block. If your generator sets lastmod to the build timestamp, a crawler accumulates a month of evidence that dates move while content does not. After that the signal is discounted across the whole domain, and regaining trust is harder than never losing it.
Warning. An honestlastmodon a third of your pages is worth more than a "fresh"lastmodon all of them. If there is nowhere to source a real modification date from, omit the tag. No signal beats a false signal.
Protocol limits: 50,000 URLs, 50 MB and the sitemap index
The protocol imposes two hard limits per file: no more than 50,000 URLs and no more than 50 MB uncompressed (52,428,800 bytes). Both apply simultaneously — a file with 30,000 very long URLs carrying hreflang annotations will hit the megabytes before it hits the record count.
A sitemap may be served gzipped, either as sitemap.xml.gz or via Content-Encoding: gzip. The 50 MB limit is measured on the uncompressed size, so compression saves bandwidth but does not raise the ceiling.
Once you exceed the limit, split the files and declare them through an index:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://example.com/sitemap-static.xml</loc>
<lastmod>2026-05-02</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemap-categories.xml</loc>
<lastmod>2026-07-10</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemap-products-1.xml.gz</loc>
<lastmod>2026-08-04T02:15:00+03:00</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemap-products-2.xml.gz</loc>
<lastmod>2026-08-04T02:15:00+03:00</lastmod>
</sitemap>
<sitemap>
<loc>https://example.com/sitemap-articles.xml</loc>
<lastmod>2026-08-05</lastmod>
</sitemap>
</sitemapindex>The index file obeys the same limits: up to 50,000 entries and up to 50 MB. Formally that caps a host at 2.5 billion URLs — nobody gets there in practice.
Index nesting
An index file cannot reference another index file. A two-level "index → indexes → sitemaps" arrangement is invalid: the engine reads the top level, finds indexes instead of sitemaps, and the nested files are never processed. If your sharding has outgrown one level, the fix is not a second level — publish several independent index files instead (robots.txt accepts as many Sitemap: lines as you need).
One more constraint: an index may only list sitemaps on the same host. A shared index across several subdomains works only when you have verified ownership of all hosts in the search console — see the host boundary discussion in subdomains versus subdirectories.
Practice. Do not run up against 50,000. Cap each shard at 20,000–45,000 URLs: you keep headroom for catalogue growth, files build and serve faster, and a generation bug costs you one shard instead of the entire sitemap.
URL rules: absolute addresses, one host, one scheme, correct escaping
Absolute URLs on the same host and scheme
Every <loc> is a full address including the protocol. Relative paths such as /catalog/ are not part of the protocol. The scheme and host must match the sitemap's own address: a sitemap at https://example.com/sitemap.xml may not list http://example.com/... or https://www.example.com/... — to a search engine those are different hosts.
This is where the classic "http URLs in an https sitemap" bug comes from. The scenario is always the same: the site moved to HTTPS, redirects were configured, but the sitemap generator keeps pulling an old base URL from a config file or from the "site address" field in the CMS. Every URL in the sitemap then costs an extra 301 hop, and crawl budget is spent on redirects instead of pages. How to diagnose that is covered in 301 versus 302 and in the broader guide to redirects.
The directory restriction
By protocol, a sitemap only covers URLs at or below its own directory: a file at https://example.com/catalog/sitemap.xml may list addresses under /catalog/ but not under /blog/. There are two legitimate ways around it — put the file in the site root (the recommended option), or declare it with a Sitemap: line in robots.txt, which makes it apply to the whole host.
XML escaping and percent-encoding
The content of <loc> is an XML value, not "just a link". Five characters must be escaped: & → &, < → <, > → >, ' → ', " → ". In practice the file is almost always broken by a bare ampersand coming from a query string.
Non-ASCII addresses are a separate story. They must be percent-encoded per RFC 3986: the string is encoded as UTF-8 first, then each byte is written as %XX. Some parsers will forgive raw non-ASCII characters, but a validator will reject the file, and some crawlers will normalise the address into a different form and end up with a duplicate.
# Wrong
<loc>https://example.com/каталог/окна/</loc>
<loc>https://example.com/catalog?a=1&b=2</loc>
<loc>/catalog/windows/</loc>
<loc>http://example.com/catalog/</loc>
# Right
<loc>https://example.com/%D0%BA%D0%B0%D1%82%D0%B0%D0%BB%D0%BE%D0%B3/%D0%BE%D0%BA%D0%BD%D0%B0/</loc>
<loc>https://example.com/catalog?a=1&b=2</loc>
<loc>https://example.com/catalog/windows/</loc>
<loc>https://example.com/catalog/</loc>
# Encode a path quickly
python3 -c "import sys,urllib.parse; print(urllib.parse.quote(sys.argv[1], safe='/:?=&'))" \
'https://example.com/каталог/окна/'Protocol extensions: image, video, news and hreflang
The base protocol describes addresses only. On top of it sit extensions — extra namespaces that add metadata. Adopt them deliberately: each one inflates the file and has to be supported by your generator.
Image sitemaps
Worth having when images are injected by scripts, hosted on a separate CDN host, or are the content itself (stock libraries, catalogues, portfolios). Of the original fields only the image location is still processed today — caption, title, licence and geolocation tags have been deprecated and are ignored.
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">
<url>
<loc>https://example.com/catalog/windows/</loc>
<image:image>
<image:loc>https://cdn.example.com/img/win-1.jpg</image:loc>
</image:image>
<image:image>
<image:loc>https://cdn.example.com/img/win-2.jpg</image:loc>
</image:image>
</url>
</urlset>Note that the images themselves may live on a different host — the same-host rule does not apply to image addresses. The cap is one thousand images per page.
Video sitemaps
Worth building if video is your primary content: streaming services, learning platforms, video blogs. The extension carries a title, description, thumbnail URL, player or content URL, duration and expiry date. For a site with one hero video on the home page the effort does not pay off — on-page structured data is enough.
News sitemaps
A separate format for publishers accepted into news services. The key constraint: the file may only contain articles published within the last two days, and everything older must be pruned out of it. It carries the publication name, language, publication date and title. Several historic tags — genres, keywords, access level — are no longer processed.
hreflang via xhtml:link
For multilingual sites, language annotations can be moved out of the HTML and into the sitemap. That is easier to maintain: edits happen in one generator instead of every template, and the <head> of each page stays lean, which also simplifies work with title, description and other meta tags.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
xmlns:xhtml="http://www.w3.org/1999/xhtml">
<url>
<loc>https://example.com/pricing</loc>
<xhtml:link rel="alternate" hreflang="de" href="https://example.com/de/pricing"/>
<xhtml:link rel="alternate" hreflang="en" href="https://example.com/pricing"/>
<xhtml:link rel="alternate" hreflang="x-default" href="https://example.com/pricing"/>
</url>
<url>
<loc>https://example.com/de/pricing</loc>
<xhtml:link rel="alternate" hreflang="de" href="https://example.com/de/pricing"/>
<xhtml:link rel="alternate" hreflang="en" href="https://example.com/pricing"/>
<xhtml:link rel="alternate" hreflang="x-default" href="https://example.com/pricing"/>
</url>
</urlset>Two rules trip people up most often. First, every entry must list all language versions, including itself. Second, the relationships must be reciprocal — if the English page points at the German one, the German page must point back. One-way links are discarded wholesale, not partially.
What belongs in a sitemap and what does not
There is one criterion and it is simple: the file should contain only URLs you want indexed as standalone pages. Formally that means three conditions at once:
- The URL returns 200 OK — not a redirect, not 404, not 403, not 5xx;
- The URL is canonical — the page's
rel="canonical"points at itself; - The URL is indexable — not blocked in robots.txt, no
noindexin the meta tag or in theX-Robots-Tagheader.
Conflicts that devalue the file
- URL blocked in robots.txt. The crawler cannot fetch the page to inspect it but sees the address listed. The result is "discovered — currently not indexed" entries and a conflict warning in the console.
- Pages carrying
noindex. A direct contradiction: the sitemap says "index this", the meta tag says "do not".noindexwins and the sitemap loses credibility. - Redirects. Every such URL is a crawler request thrown away. On large sites that is a measurable slice of crawl budget.
- 404 and 410. A clear sign the sitemap was never rebuilt after content was removed.
- Non-canonical variants. URLs with campaign parameters, session identifiers, sort and filter states, www and non-www forms. They all collapse into a canonical address and have no business being listed.
- Utility pages. Cart, checkout, account area, internal search results, thank-you pages, test and staging addresses.
- Pagination and faceted filters. A judgement call: include them if the pages are self-canonical and carry value, exclude them if they collapse onto page one.
Diagnostic rule. The "indexed / submitted" ratio is only meaningful if the sitemap holds a clean list. Dump everything into it and you get a nominal 20% with no way to separate a content-quality problem from your own noise.
Splitting by content type: turning a sitemap into a diagnostic tool
A single 40,000-URL sitemap.xml answers "how many are indexed" and fails to answer "which ones are not". Splitting solves that for free: webmaster consoles report statistics per file.
A practical layout for a store: sitemap-static.xml (a dozen landing pages), sitemap-categories.xml, sitemap-products-N.xml, sitemap-articles.xml. If the report shows categories at 98%, articles at 90% and products at 34%, you have localised the problem to product pages in under a minute instead of guessing across the whole site.
The second axis is partitioning within a type. For products, splitting by identifier range or by date added works well: the "fresh" shard is rebuilt often, the historic ones rarely, and lastmod in the index file starts carrying real information.
| Site size and type | File layout | How to update |
|---|---|---|
| Landing page, brochure site — up to ~100 URLs | One static sitemap.xml in the root | By hand or at build time; rebuild whenever pages are added |
| Corporate site, blog — 100–5,000 URLs | One sitemap.xml, or an index plus two files: pages and articles | CMS plugin or build-time generation; a daily job is enough |
| Online store — 5,000–50,000 URLs | Index plus a split by type: categories, products, articles, static | Dynamic generation written to files on a schedule; the product file more often than the rest |
| Large catalogue, marketplace — 50,000+ URLs | Index plus sharding by type and partition, 20,000–45,000 URLs each | Dynamic generation from the database only, incrementally: rebuild the changed shard, not the whole set |
| News site, media | Regular index plus a separate news sitemap (last two days only) | News file on every publication; the general set on a schedule |
| Multilingual site | Index plus one file per language, hreflang via xhtml:link | Generate from a single data source so alternate sets never drift apart between languages |
| Site with subdomains | Its own sitemap and its own robots.txt on every host | Independently; a shared index only when ownership of all hosts is verified |
How to produce the file: CMS plugin, crawler, build script, dynamic generation
CMS plugin or built-in module
The fastest route. Mainstream platforms build a sitemap either out of the box or through an SEO plugin. There is one significant downside: by default everything ends up in the file — tags, date archives, author pages, media attachments. Configuring exclusions is a required step, not an option. After installing a plugin, open the resulting file and strike out anything that should not be a standalone page in search.
Crawler-based generators
A desktop tool or online service walks the site like a crawler and exports a static file. Upside: the crawler sees real availability and will not list what it could not reach. Downsides: it is a snapshot that goes stale; it will not find orphan pages (which are often the whole reason for building a sitemap); and on large sites the crawl takes hours. The sensible range is sites up to a few tens of thousands of URLs with infrequent changes.
Build-time generation
For static site generators and SSG frameworks the sitemap is assembled during the build, from the same source as the pages themselves. The URL list matches what was actually deployed, by construction. The ideal option for documentation, blogs and content projects.
Dynamic generation from a database
For a large site there is no alternative. Only database-driven generation guarantees that the sitemap reflects the current state: unpublish a product and it disappears from the file on the next cycle. The key nuance is not to render the sitemap on every request — a 200,000-row query plus XML assembly is an expensive operation, and crawlers are not the only clients asking for it. The correct pattern is to build files on a schedule and serve them as static assets.
<?php
// build-sitemap.php — run on a schedule, output is served as a static file
$base = 'https://example.com';
$dir = '/var/www/example.com/public';
$max = 45000; // headroom below the 50,000 limit
$part = 1;
$n = 0;
$sql = "SELECT slug, updated_at
FROM products
WHERE is_published = 1
AND is_indexable = 1
AND canonical_id IS NULL
ORDER BY id";
$open = function (int $p) use ($dir) {
$f = fopen("$dir/sitemap-products-$p.xml", 'w');
fwrite($f, '<?xml version="1.0" encoding="UTF-8"?>' . "\n");
fwrite($f, '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">' . "\n");
return $f;
};
$out = $open($part);
foreach ($pdo->query($sql, PDO::FETCH_ASSOC) as $row) {
if ($n >= $max) { // close the shard, open the next one
fwrite($out, "</urlset>\n");
fclose($out);
$out = $open(++$part);
$n = 0;
}
$path = '/product/' . rawurlencode($row['slug']);
$loc = htmlspecialchars($base . $path, ENT_XML1 | ENT_QUOTES, 'UTF-8');
$mod = date('Y-m-d', strtotime($row['updated_at'])); // content date, not build time
fwrite($out, " <url><loc>$loc</loc><lastmod>$mod</lastmod></url>\n");
$n++;
}
fwrite($out, "</urlset>\n");
fclose($out);Three things here matter more than the code itself: the query filters on "published", "indexable" and "canonical"; the path goes through rawurlencode and the final URL through XML escaping; and lastmod comes from a content field rather than from time().
How to declare a sitemap: robots.txt, search consoles, IndexNow
The Sitemap line in robots.txt
The most universal method: any crawler that supports the protocol finds the sitemap without any registration.
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /search
Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-news.xmlThe rules: the address must be absolute; the line belongs to no User-agent group and applies globally, which is why it is conventionally placed at the end of the file; there may be several lines — for instance one for the main index and one for a news sitemap. The directive name is case-insensitive. Full syntax and precedence rules are in the robots.txt guide.
Search consoles
In Google Search Console a sitemap is added in the Sitemaps section of a verified property — submit the index file and the children are picked up automatically. Bing Webmaster Tools works the same way, and Yandex.Webmaster has an equivalent section whose documentation allows up to two weeks for processing a newly added file, so there is no reason to panic on day three.
The value of consoles is not the submission itself (robots.txt achieves the same) but the reporting: processing status, last read date, number of URLs discovered, and a list of errors with specific line numbers.
Ping endpoints and IndexNow
Here is an update most guides still omit. The classic ping endpoints that people hit after publishing for years have lost their role: Google removed any mention of such an endpoint from its documentation, and Bing dropped its own in favour of a different mechanism. "Ping after deploy" is no longer a meaningful step.
The current fast-signal option is the IndexNow protocol: you place a key file on the site and send an HTTP request listing the changed addresses. Bing, Yandex and several smaller engines accept it; Google does not participate. The mental model is straightforward — the sitemap is the baseline channel everyone reads, IndexNow is an additional fast signal for the engines that accept it. Neither replaces the other.
How often to rebuild the file
Driven by actual changes, not by the calendar. A content site: nightly. A store with a stable assortment: daily, with the product shard rebuilt more often during bulk imports. A news site: the news sitemap on every publication. A static brochure site: on deploy. If nothing changed, do not rebuild — that only corrupts lastmod.
How to validate a sitemap
Response headers and XML validity
The first thing to confirm is that the file is served directly, with a 200 and the right content type. A correct Content-Type is application/xml or text/xml. If you see text/html, a CMS or SPA router almost certainly intercepted the request and returned an HTML template instead of the file.
# 1. Headers: status code, content type, compression
curl -sI https://example.com/sitemap.xml \
| grep -iE 'HTTP/|content-type|content-encoding|content-length|x-robots-tag'
# 2. XML validity (silence means no errors)
curl -s https://example.com/sitemap.xml | xmllint --noout -
# 3. For a gzipped sitemap
curl -s https://example.com/sitemap-products-1.xml.gz | gunzip | xmllint --noout -
# 4. Validate against the official protocol XSD schema
curl -sO https://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd
curl -s https://example.com/sitemap.xml -o sitemap.xml
xmllint --noout --schema sitemap.xsd sitemap.xml
# 5. A BOM at the start of the file — a classic cause of "parsing error"
curl -s https://example.com/sitemap.xml | head -c 3 | xxd
# efbb bf in the output means a BOM is present and must be removedSpot-checking status codes
Valid XML says nothing about whether the listed pages are alive. A reasonable compromise between "check everything" and "check nothing" is a random sample of fifty addresses. If even one redirect or 404 shows up in it, the problem is almost certainly systemic.
# Status codes for 50 random URLs from the sitemap, 8 requests in parallel
curl -s https://example.com/sitemap.xml \
| grep -oE '<loc>[^<]+</loc>' \
| sed -E 's#</?loc>##g' \
| shuf -n 50 \
| xargs -P 8 -I{} curl -s -o /dev/null -w '%{http_code} %{url_effective}\n' {} \
| sort | uniq -c | sort -rn
# Total URL count, and whether any http addresses slipped into an https site
curl -s https://example.com/sitemap.xml | grep -c '<loc>'
curl -s https://example.com/sitemap.xml | grep -c '<loc>http://'The expected result of the first command is a single line reading 50 200 …. Any 301, 302, 404 or 5xx in the output means the generator is out of sync with the real state of the site. Status code reference: HTTP status codes explained.
Console reports
Watch three numbers per file: URLs discovered, URLs indexed, errors. The gap between the first two is your real measure of content quality and technical accessibility. "Discovered — currently not indexed" and "Crawled — currently not indexed" mean different things: in the first case the crawler never fetched the page, in the second it fetched it and decided against it.
enterno.io tools
- SEO audit — checks that a sitemap exists and is reachable, and finds conflicts between the sitemap, robots.txt and meta directives.
- robots.txt checker — is the
Sitemapdirective present, is the address absolute, are the listed sections blocked from crawling. - Broken link checker — the fastest way to find 404s among the URLs you declared to search engines.
- Redirect checker — run addresses from the sitemap through it to catch extra hops and chains.
- HTTP header analyser —
Content-Type,Content-EncodingandX-Robots-Tagof the sitemap file itself.
Common mistakes: symptom, cause, check, fix
| Symptom | Cause | How to check | Fix |
|---|---|---|---|
| "Sitemap could not be fetched" in the console | The file returns 404, a redirect, or is behind basic auth | curl -sI against the sitemap address | Serve a 200 directly; remove auth from the sitemap path |
| "Parsing error" | BOM or whitespace before the XML declaration, an unescaped &, a truncated file | head -c 3 | xxd, then xmllint --noout | Strip the BOM, escape special characters, verify generation runs to completion |
| The sitemap renders as a normal page | A CMS or SPA router intercepts the request and returns Content-Type: text/html | curl -sI plus a look at the first lines of the body | Serve a static file ahead of the router, or set the content type explicitly |
| 50,000 submitted, 1,200 indexed | The sitemap is full of noise: duplicates, filters, utility and thin pages | Status-code sample plus a check against rel="canonical" | Prune to canonical pages, split by type and measure each separately |
| "Sitemap contains URLs blocked by robots.txt" | Conflicting directives: the address is declared and simultaneously blocked | robots.txt checker | Decide which is true: drop the URLs or open the section for crawling |
| http addresses in the sitemap of an https site | A hardcoded base URL in config or CMS settings | grep -c '<loc>http://' over the file | Fix the base URL and rebuild every shard |
| Sitemap lives in a subfolder but covers the whole site | The protocol's directory restriction | Compare the file path with the paths inside <loc> | Move it to the root, or declare it with a Sitemap: line in robots.txt |
Every URL has lastmod = today | The generator writes the build timestamp | Open the file and look at the spread of dates | Take the date from a content modification field; if there is none, omit the tag |
| Addresses removed a year ago are still listed | A static file nobody rebuilds | Spot-check status codes | Move generation to a schedule or into the build pipeline |
| Non-ASCII addresses written raw | The generator does not encode the path | Validation against the XSD schema | rawurlencode for the path, then XML-escape the final URL |
| The index is processed but child files are not | The index points at another index, or at sitemaps on a different host | Inspect the root element of the child files | Flatten to one level; declare multiple indexes as separate lines in robots.txt |
| File exceeds 50 MB or 50,000 URLs | No sharding | grep -c '<loc>' and the file size | Split into shards of 20,000–45,000 URLs and build an index |
Frequently asked questions
Does a small site need a sitemap.xml?
It is not mandatory for any site. For a few dozen pages with solid internal linking the effect is close to zero: the crawler covers everything anyway. But the file costs nothing, and your search console gains a clear "submitted versus indexed" figure — that alone justifies building one.
Does a sitemap affect rankings?
No. Neither the file itself nor priority nor changefreq takes part in ranking. A sitemap influences discovery and crawl ordering only. Promises of "we will set up your sitemap and your rankings will rise" are marketing, not search mechanics.
Where should the file live and what should it be called?
The name is arbitrary — sitemap.xml is simply the common convention. Location matters more: by protocol a sitemap only covers its own directory and everything below it, so the site root is the one place where the question never arises. A file in a subfolder is allowed, but then it must be declared in robots.txt.
What do I do when I have more than 50,000 URLs?
Split into several files and build an index. Do not try to squeeze under the limit by dropping pages: the limit exists for manageability, not to cap the size of your site. A sensible shard size is 20,000–45,000 URLs.
Should I fill in changefreq and priority?
There is no practical benefit. Google ignores them and other crawlers treat them as a weak hint at best. They do no harm beyond a few extra bytes — but if you are writing a generator from scratch, you can safely skip them and spend that time on an honest lastmod.
How long does indexing take after submitting a sitemap?
There are no guaranteed timelines. Processing the file itself usually takes hours to days; Yandex documentation allows up to two weeks. Indexing of specific pages depends on their quality, the site's authority and its crawl budget — the sitemap has little influence on that timeline.
Can I list redirecting or noindex pages?
Technically yes, practically no. A redirect wastes a crawler request and noindex creates a direct signal conflict. Occasionally a redirect is added deliberately so a crawler learns about a migration faster — but that is a one-off measure followed by cleanup, not a permanent state of the file.
Does a subdomain need its own sitemap?
Yes. A subdomain is a separate host with its own robots.txt and its own sitemap. You cannot list its URLs in the main domain's sitemap unless you have verified ownership of both hosts in the search console.
Checklist
- The file opens at its direct address, returns 200 and
Content-Type: application/xml. - Encoding is UTF-8, no BOM, no whitespace or blank lines before the XML declaration.
- The
http://www.sitemaps.org/schemas/sitemap/0.9namespace is present and unmodified. - Every
<loc>is an absolute URL on the same scheme and host as the file itself. - Ampersands and other special characters are escaped; non-ASCII paths are percent-encoded.
- The sitemap lists only canonical, indexable URLs that return 200.
- No overlap with
Disallowrules in robots.txt or with pages carryingnoindex. - No file exceeds 50,000 URLs or 50 MB uncompressed.
- The index file references only regular sitemaps on the same host, with no nested indexes.
lastmodreflects a real content change, not the build timestamp.- The sitemap is split by content type so console reports expose the weak sections.
- robots.txt contains a
Sitemap:line with an absolute address. - The file is submitted in Google Search Console and Bing Webmaster Tools, and its processing status has been checked.
- Generation is automated — a schedule or a build step, not manual edits.
- A sample of 50 random URLs is periodically checked for status codes.