Skip to content

sitemap.xml: Definition and Use Cases

TL;DR:

sitemap.xml is an XML file listing every canonical URL for search engines. Contains loc, lastmod, changefreq, priority. Limit: 50,000 URLs / 50 MB per file. For large sites — sitemap-index referencing partitioned sitemaps.

Check your site →

What is sitemap.xml

sitemap.xml is an XML file listing every canonical URL for search engines. Contains loc, lastmod, changefreq, priority. Limit: 50,000 URLs / 50 MB per file. For large sites — sitemap-index referencing partitioned sitemaps.

How to Create a Sitemap.xml File

Creating a sitemap.xml file is essential for optimizing your website's visibility in search engines. Below are steps to generate a sitemap using various methods:

  • Manual Creation: You can create a sitemap manually using a text editor. Ensure each URL is wrapped in <url> tags:
  • <?xml version='1.0' encoding='UTF-8'?>
    <urlset xmlns='http://www.sitemaps.org/schemas/sitemap-image/1.1'>
      <url>
        <loc>http://example.com/page1</loc>
        <lastmod>2023-10-01</lastmod>
        <changefreq>monthly</changefreq>
        <priority>0.8</priority>
      </url>
    </urlset>
  • Using CMS Plugins: If you’re using a Content Management System (CMS) like WordPress, plugins like Yoast SEO or Google XML Sitemaps can automate sitemap generation.
  • Command-Line Tools: For developers, tools like xml-sitemaps-generator can be run via command line:
  • npm install -g sitemap-generator-cli
    sitemap-generator https://example.com --output sitemap.xml

Once created, upload the sitemap.xml to your website's root directory. Verify its presence by navigating to http://example.com/sitemap.xml.

Understanding Sitemap.xml Elements

A sitemap.xml file contains several key elements that provide search engines with information about your website's pages. Here’s a breakdown of the most important elements:

  • <loc>: This tag specifies the URL of the page. It is a required field in every <url> entry.
  • <lastmod>: Indicates the last modification date of the URL. This is optional but helps search engines understand how often to crawl the page.
  • <changefreq>: Suggests how frequently the page is likely to change (e.g., always, hourly, daily, weekly, monthly, yearly, never). This is also optional.
  • <priority>: Assigns a priority to the URL relative to other URLs on your site. Values range from 0.0 to 1.0, with 1.0 being the highest priority. This is optional as well.

Understanding these elements allows for better control over how search engines interpret your website structure, which in turn can influence indexing and ranking.

Common Sitemap.xml Issues and Solutions

While implementing sitemap.xml, you may encounter several common issues that can affect your site's SEO performance. Here are some typical problems and their solutions:

  • Incorrect URL Formatting: Ensure that all URLs are properly formatted and accessible. A common mistake is including URLs that return a 404 error. Use tools like Google Search Console to check for crawl errors.
  • Exceeding URL Limits: If your sitemap exceeds 50,000 URLs or 50 MB, you need to create a sitemap index file to reference multiple sitemaps. Ensure that each sitemap adheres to the limits.
  • Not Submitting to Search Engines: After creating your sitemap, submit it to search engines via their webmaster tools (e.g., Google Search Console, Bing Webmaster Tools) to ensure they are aware of its existence.
  • Not Updating Regularly: If your content changes frequently, automate the sitemap update process. Use CMS plugins or scripts that regenerate the sitemap whenever content is updated.

By addressing these issues, you can optimize your sitemap.xml file for better search engine visibility and performance.

robots.txt ParsingFull Allow/Disallow directive parsing
URL TestCheck if a specific URL is allowed for bot
Sitemap LinksAll Sitemap: directives in the file
AI CrawlersGPTBot, ClaudeBot, and other AI bots

Why teams trust us

Live
robots.txt check
UA
any User-Agent
Sitemap
sitemap links
Free
no signup

How it works

1

Enter site URL

2

Parse robots.txt

3

Check crawl rules

Why check robots.txt?

robots.txt controls which pages search bots can see. Incorrect directives can accidentally block the entire site from indexing or expose administrative sections.

Full Parsing

Parse robots.txt per RFC 9309: all User-agent, Allow/Disallow, Crawl-delay, Sitemap.

URL Tester

Enter a specific URL and User-agent — find out if it's allowed for that bot.

AI Crawlers

Automatically show status for GPTBot, ClaudeBot, PerplexityBot, Googlebot.

Sitemap List

All Sitemap: directives in one place with quick links for verification.

Who uses this

SEO

crawl directive audit

Developers

post-deploy check

Marketers

indexation control

Site owners

block unwanted crawlers

Common Mistakes

Disallow: / for entire siteThis blocks the entire site from indexing. Check robots.txt after every change.
Blocking AI without understandingBlocking GPTBot removes your site from ChatGPT and Perplexity citations.
Not specifying SitemapWithout a Sitemap: directive, bots must guess the sitemap URL. Always specify explicitly.
Conflicting rulesAllow and Disallow on the same URLs for different User-agents create unpredictable behavior.

Best Practices

Test after every changeOne wrong character in robots.txt can block an entire section from indexing.
Use * carefullyUser-agent: * applies to all bots, including AI crawlers.
Always specify SitemapSitemap: https://example.com/sitemap.xml helps bots find all pages.
Verify with Google Search ConsoleGSC shows how Google sees your robots.txt, including parsing errors.

Get more with a free account

Robots.txt check history and change monitoring for your site.

Sign up free

Learn more

Frequently Asked Questions

Do I need sitemap.xml?

If you work with web infrastructure — yes. See description above.

Try the live tool that powered this guide

Free plan — 10 monitors, checks every 5 min, no card required. Upgrade for 1-minute interval and multi-region monitoring.