What Is Website Indexing: How It Works and How to Check

In short. Website indexing means storing your pages in a search engine's own database after a robot downloaded and processed them. Search queries that database, not the live web: a page missing from the index cannot appear in results. Indexing happens after crawling and before ranking, so «indexed» and «ranking well» are different states.

What website indexing is in plain terms

When you type a query, the search engine does not go out and poll billions of sites. It looks into a database it built in advance — the index. Structurally it works like the index at the back of a book, only inverted: not "page 148 contains the words A, B, C" but "the word A appears on pages 12, 148, 903". This is called an inverted index, and it is why a result set comes back in milliseconds without a single site being contacted.

Indexing is the process that turns a downloaded page into entries in that structure. Along the way the search engine does a fair amount of work:

• parses the HTML and separates the main content from navigation, footer and ad blocks;
• extracts structure — headings, lists, tables, structured data;
• detects the language and the general topic of the page;
• collects links so new addresses can be queued for crawling;
• compares the page with pages it already knows: if it is a duplicate, only one address from the group is stored;
• keeps operational data — status code, crawl date, response headers, a cached copy.

The outcome is binary: the page either enters the index and can compete in search results, or it does not. The second outcome is a normal decision, not necessarily a fault on your side. Not everything that gets downloaded gets indexed, and that is deliberate economy — the database has to be stored, refreshed and served.

Which leads to the first practical consequence. "Why is my page not in search?" is almost never answered by "wait longer". It is answered by working out which stage the page is stuck at: it was never discovered, it could not be downloaded, it was downloaded and discarded, or it is indexed but losing to competitors for that specific query. Four different problems, four different fixes. If the whole site is missing from search rather than a single page, start with the dedicated walkthrough on what to do when a site is not in search — this article explains the mechanism, not the emergency.

Web indexing versus crawling: two different things

This is the single biggest source of confusion in the topic, and it costs people weeks. "Web indexing" colloquially covers everything a search robot does, but inside it there are two fundamentally different actions.

Crawling means the robot makes an HTTP request to an address and downloads the response. That is all. The only question answered at this stage is whether the content could be retrieved. The output is a status code and a response body.

Indexing means the search engine processes what it downloaded and decides whether to store the document. The output is yes or no.

The two stages are independent enough that all four combinations occur in practice.

State	What happened	How it looks from outside
Not crawled, not indexed	The robot does not know the address, or has not reached it yet	The page is nowhere: not in the console, not in results
Crawled, not indexed	The page was downloaded but judged unnecessary, or blocked by a meta tag	The console shows a crawl date and a "not indexed" status
Not crawled, indexed	Crawling is blocked in robots.txt, but the address is known from links	A bare URL in results with no real title or description
Crawled, indexed	The normal working case	The page participates in ranking

The third row is why people block pages in robots.txt for years and keep wondering why they still show up in search. The robots.txt file governs crawling, not the index. It tells the robot "do not download this", and the robot obeys — but it already knows the address from internal and external links, and it may show that bare URL with no content behind it. The Robots Exclusion Protocol is specified in RFC 9309, and it says nothing about removing anything from an index — only about access to fetching.

Rule. Disallow in robots.txt blocks downloading, not indexing. To drop a page from the index the robot must be able to fetch it and see the prohibition inside: a robots meta tag or an X-Robots-Tag header. A page that is blocked in robots.txt and carries noindex inside will stay indexed forever — the robot can never read your noindex.

How a page gets into the index, step by step

Here is the chain broken down. Knowing exactly where the boundary between steps sits saves more time than any checklist.

1. Discovery

The robot has to learn the URL from somewhere. Sources: links on pages it already knows (internal and external), the sitemap.xml file, redirects from old addresses, a manual re-crawl request in a webmaster console, instant-notification protocols such as IndexNow. A page with no inbound link and no sitemap entry is an orphan: technically fine, practically invisible.

2. Queueing

A discovered address goes into a queue. Its position and crawl frequency depend on how important the page looks and how much load the server can take. This is the one stage you influence only indirectly — through internal links, the sitemap and response speed.

3. Crawling

Before requesting anything the robot reads the domain's robots.txt. If the address is disallowed, no fetch happens. If it is allowed, the robot makes the request and gets a status code, and the code decides what follows: 200 means work with the body; 3xx means follow the redirect; 404 or 410 means the document is gone and the address will eventually drop out of the index; 5xx means the server is broken, so the robot slows down and comes back later.

4. Rendering

The source HTML is parsed immediately. If a meaningful part of the content only appears after JavaScript executes, the page goes into a separate rendering queue — an extra stage that can be deferred. The practical conclusion: the more meaning lives in the raw HTML, the fewer dependencies and failure points you have. You can see exactly what arrives before scripts run with plain curl, which does not execute JavaScript.

5. Analysis and canonicalisation

The document is compared with known ones. Similar pages are grouped and one canonical address is chosen for the group — that is the address that gets indexed. Your rel="canonical" is taken into account, but as a hint rather than a command: if it contradicts other signals such as internal links, the sitemap or redirects, the search engine may pick a different address.

6. Storing

The page takes its place in the database. From this moment it can be shown for queries.

7. Ranking

A separate process that runs at query time and has nothing to do with indexing.

# What the robot gets on the first pass: status code and headers
curl -sSIL https://example.com/page/ | grep -iE '^HTTP/|^location:|^x-robots-tag:'

# Raw HTML before scripts run — canonical and indexing directives
curl -sS https://example.com/page/ | grep -iE 'rel="canonical"|name="robots"|name="googlebot"'

Indexing versus ranking: indexed is not the same as visible

Indexing is a property of a document: it is either in the database or it is not. Ranking is a property of a document–query pair: the same page can be third for one query, outside the first hundred for another and absent for a third. There is no "level of indexing" or "indexing quality" — you cannot be halfway into a database.

This produces the most common disappointment in SEO. Someone finally gets a page indexed, sees it marked as indexed in the console, and gets no visits at all. Nothing is broken. Indexing is a necessary condition for traffic and nowhere near a sufficient one. After it, the page competes for specific queries against thousands of other documents, and mere presence in a database wins nothing.

Do not merge two questions. "Is the page indexed?" is checked in a webmaster console and the answer is binary. "Does the page bring traffic?" is checked through impressions and positions and the answer is a number. The first is fixed with technique: availability, status codes, directives, duplicates. The second is fixed with content, demand and competition. Technical edits do not lift a page that is already indexed cleanly.

Check them in that order: presence in the index first, impressions second. If there are no impressions but the page is indexed, the technical part is closed and the remaining work is about demand and content.

How to check if your site is indexed

There are three sources of truth, and they answer different questions. The most accurate is the webmaster console of the search engine in question, because it reports data from the same system that made the decision.

Google Search Console

Two levels of detail. The summary indexing report shows how many known addresses ended up indexed and groups the rest by reason. The URL inspection tool works on a single address: whether it is known, when it was crawled, which canonical the system selected (it can differ from the one you declared) and whether anything blocks it. The same tool shows the rendered version of the page — the only practical way to see a JavaScript-built page through the search engine's eyes.

The crawl stats report deserves separate attention: it shows how many requests the robot makes, which status codes it receives and how long it waits for a response. A spike of errors or a rising response time there explains slow indexing better than any guesswork.

Bing Webmaster Tools and regional consoles

Bing Webmaster Tools follows the same logic — a site-wide indexing report plus per-URL inspection — and additionally supports IndexNow submissions. If a meaningful share of your audience is in a market where a regional engine leads, register there too: Yandex Webmaster for Russian-speaking markets, Naver for South Korea, Baidu for mainland China. Their reports are the only authoritative answer about their own index, and their decisions do not have to match Google's.

The site: operator and why its number lies

A query like site:example.com lists addresses of the domain that search knows about. As a quick qualitative check — "is anything indexed at all?" — it works. As a counter it does not.

The result count for a site: query is an estimate, not a report. It fluctuates between identical queries, depends on region and personalisation, and consistently disagrees with the numbers in a webmaster console. Do not make decisions on "it said 1200 yesterday and 900 today" — most likely nothing changed at all. For quantitative conclusions the console is the only source.

The operator does have one good use: targeted probing. Combining site: with a URL fragment quickly answers "did a junk section leak into the index?" — for example site:example.com inurl:search shows whether internal search result pages got in.

Server-side checks: what you actually serve

Consoles show the search engine's decision. But half of all indexing problems are your site serving the robot something you did not intend, and that is only visible from the server side. The minimum set of checks:

• The status code. A page that looks fine in a browser but returns 404 or 503 is a classic after migrations and cache misconfigurations.
• The X-Robots-Tag header. The most invisible prohibition there is: absent from the HTML, invisible in the browser, and a complete block on indexing. Inspect the headers of any page with the HTTP header analyser.
• robots.txt. One stray Disallow closes an entire section. Parse the rules and test whether a specific address matches with the robots.txt checker.
• Meta tags and canonical in the raw HTML — before JavaScript runs.
• Redirects. A multi-hop chain, a loop, or a redirect to the homepage instead of the target page is a common reason content never reaches the index; trace it with the redirect tracer.

All of these run site-wide in the SEO audit: it walks the pages and collects status codes, indexing directives, canonical addresses and duplicate titles into one report. Start there, then bring in the single-purpose tools once you have a hypothesis.

# Full picture for one address: every redirect hop and every directive
curl -sSIL -A 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)' \
  https://example.com/page/ | grep -iE '^HTTP/|^location:|^x-robots-tag:|^content-type:'

# robots.txt exactly as the robot reads it
curl -sS https://example.com/robots.txt

What blocks indexing

Obstacles fall into two classes: explicit prohibitions you set yourself (sometimes without knowing it) and indirect ones, where the system decides the document is not worth the storage. The first class is fixed in minutes; the second requires work on the content.

Mechanism	Where it lives	What it does	Can the address still be indexed	Typical mistake
Disallow in robots.txt	File at the domain root	Blocks fetching	Yes — as a bare URL discovered via links	Used as a way to remove a page from search
<meta name="robots" content="noindex">	Page head	Blocks indexing of the content	No, provided the robot can fetch the page	Combined with Disallow, so it is never read
X-Robots-Tag	HTTP response header	The same, for any file type	No	Left over from staging and shipped to production
rel="canonical"	Page head or response header	Names the preferred address of a duplicate group	Usually the named canonical is indexed instead	Every page points at the homepage
301 / 302 redirect	Server response	Sends the robot to another address	The redirect target is indexed	Chains and loops, all 404s redirected to the homepage
404 / 410	Server response	Declares the document gone	No, the address drops out over time	A working page returns 404 due to a routing bug
5xx	Server response	Declares a failure	Crawling slows; sustained errors drop the page	The server returns 503 to robots under load and nobody notices
Authentication required	401 / 403 response	Denies access	No	A section was password-protected "temporarily" and forgotten

robots.txt

The file must sit at the root of the domain and applies only to that host and port. A subdomain is a separate site with its own file. The syntax is defined in RFC 9309: groups of rules per user agent, Allow and Disallow directives, and on conflict the longer (more specific) rule wins.

# Rules for every robot
User-agent: *
# internal search produces an unbounded number of combinations
Disallow: /search/
# private areas
Disallow: /cart/
Disallow: /account/
# sort parameters generate duplicates of the same listing
Disallow: /*?sort=
# never block the CSS and JS needed to render the page
Allow: /*.css$
Allow: /*.js$

Sitemap: https://example.com/sitemap.xml

The two most expensive mistakes in this file are blocking /assets/ with the stylesheets and scripts (the robot then renders a broken page) and a Disallow: / that travelled from staging into a release. The second one closes the whole site and is usually discovered by a traffic collapse. A full walkthrough of the syntax and working templates is in the robots.txt guide.

The robots meta tag and the X-Robots-Tag header

Unlike robots.txt, these are real indexing prohibitions. The meta tag works for HTML pages; the header works for anything, including PDFs, images and data exports.

<!-- Do not index this page, but do follow its links -->
<meta name="robots" content="noindex, follow">

<!-- Preferred address for a duplicate group -->
<link rel="canonical" href="https://example.com/page/">

# nginx: keep documents and exports out of the index
location ~* \.(pdf|docx?|xlsx?|csv)$ {
    add_header X-Robots-Tag "noindex, nofollow" always;
}

# Apache: same thing
<FilesMatch "\.(pdf|docx?|xlsx?|csv)$">
    Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

Check X-Robots-Tag on production after every release. It is the least visible indexing block in existence: not in the HTML, not visible in the browser, not shown by "view source". A header set on a staging box and shipped to production inside a config can wipe a site out of search, and nobody works out why until someone looks at the response headers.

Duplicates and low-value pages

This is the indirect class. If a site can generate many near-identical addresses — listings with sort and filter parameters, pagination with no meaningful differences, print versions, one product reachable by several paths — the search engine groups them and keeps one. The rest are downloaded but never stored: formally "crawled, currently not indexed".

The same bucket holds pages with no content of their own: empty categories, auto-generated tags holding a single item, boilerplate texts that differ only by city name. The refusal here is not technical, and rewriting title tags does not cure it.

How to get your site indexed: what you actually control

You cannot control indexing directly — the search engine decides. What you can do is remove obstacles and make the robot's job easier. In order from cheapest to most expensive.

Remove explicit prohibitions

The first step is always the same: make sure you have not blocked indexing yourself. Check robots.txt, meta tags, response headers, status codes and redirect chains on the live domain — not on a local copy. It takes minutes and closes a noticeable share of cases.

Provide a sitemap

The sitemap.xml file is a list of the addresses you consider worth indexing. It guarantees nothing, but it speeds up discovery and, more usefully, acts as a reference set: the gap between the number of addresses in the sitemap and the number of indexed pages is a ready-made diagnostic.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/page/</loc>
    <lastmod>2026-01-15</lastmod>
  </url>
</urlset>

Two rules are broken more often than any others: a sitemap must contain only canonical addresses that return 200, and lastmod must be a real modification date. A sitemap where every page changed today devalues the signal entirely. Formats, limits and sitemap index files are covered in the sitemap.xml guide.

Link pages together

Internal linking is the main discovery mechanism and the main way the robot's attention is distributed. A page that cannot be reached by following links from navigation or from another article will be crawled less often and with lower priority, even if it sits in the sitemap. A practical rule: if you cannot reach a document from the homepage in a reasonable number of clicks, the robot treats it as secondary.

While you are there, make sure internal links do not point at dead addresses or redirects — the robot spends the same resource on those as on useful pages. A site-wide sweep is what the broken link checker does.

Request a re-crawl

Webmaster consoles let you push a specific address into the queue. On top of that there is IndexNow, a protocol a site uses to notify participating search engines that an address changed. Both speed up the visit; neither makes the indexing decision for the engine.

Keep the server stable

Slow responses and intermittent 5xx errors directly reduce the crawl rate: the robot protects other people's servers and, when it collects errors, comes back less often. The nastiest variant is a site that answers fine from a browser but fails under load or for one specific user agent. Response time is measurable with the speed test, but irregular failures can only be caught by continuous monitoring — a one-off check shows one second in time, while crawling runs around the clock.

Crawl budget: when to care and when not to

Crawl budget is the informal name for how many of your pages a robot is willing to download over a period. It comes from two limits: how much your server can take without hurting users, and how much the engine wants to spend on your site at all.

For most sites crawl budget is not a problem that exists. A site with a few hundred or a couple of thousand pages gets crawled in full without any optimisation. If your pages are not being indexed and the site is small, the cause is almost certainly prohibitions, duplicates or content — not budget. Starting the investigation with crawl budget in that situation means spending a week on the wrong hypothesis.

The problem becomes real when a site can generate far more addresses than it has content. The usual generators:

• Faceted navigation. Five filters with five values each produce thousands of addresses over the same set of products.
• Sort and display parameters. One listing turns into a dozen addresses.
• Internal search. Crawlable result pages are an unbounded space.
• Calendars and archives that link to any month of any year.
• Session identifiers and tracking tags in URLs when they leak into links.

The symptom of a genuine budget problem: new pages take a long time to appear in the index while the logs show the robot crawling actively — just across junk addresses. You confirm this from server logs, not from intuition.

# Which status codes the Google robot receives over the period
awk '/Googlebot/ {print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

# The addresses the robot fetches most often
awk '/Googlebot/ {print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

# How many robot requests go to parameterised addresses
awk '/Googlebot/ {print $7}' /var/log/nginx/access.log | grep -c '?'

If the top rows of the second report are filter and sort pages, the budget really is going to the wrong place. The fix: block the combination generators from crawling, collapse duplicates with canonical addresses, remove redirect chains and stop linking internally to dead addresses.

Frequently asked questions

How long does it take for a new page to be indexed?

There is no guaranteed timeframe, and any specific answer in days would be invented. The speed depends on how often the robot visits the site at all, how easily the page is discovered through links and how substantial it looks. The controllable part is discovery: a link from a frequently crawled page, an entry in the sitemap and a re-crawl request shorten the wait. Nobody, however, can guarantee indexing.

The page is "crawled, currently not indexed" — what now?

That status means there is no technical obstacle: the robot arrived, fetched, read and decided not to store the document. Checking for prohibitions is pointless — there are none. Look at two things instead: whether the page is a duplicate of another one (in which case its canonical twin is indexed) and whether it has content of its own that differs from neighbouring pages. Technical edits do not change this status.

Should service pages be blocked from indexing?

Yes, but with the right instrument. Carts, account areas, internal search result pages and technical sections are reasonable to exclude: they give a searcher nothing and blur what the site is about. For pages already in the index you need noindex with crawling left open — that is the only way the robot can see the prohibition. robots.txt is for things that are not in the index yet and should never get there.

Does site speed affect indexing?

Indirectly, and noticeably. Server response time caps the crawl rate: the slower the server, the fewer pages the robot gets through per visit. On a small site this is invisible; on a large one it becomes the bottleneck. Intermittent errors are worse still — with sustained 5xx responses the robot reduces how often it visits.

Does content loaded by JavaScript get indexed?

It can be, but it is an extra stage with an extra queue and extra failure points. If the main text, the heading and the links are present in the raw HTML, you do not depend on rendering at all. To see what exists before scripts run, request the page with curl: it returns exactly the response the robot receives on its first pass.

How do I remove a page from the index?

There is one reliable way: leave the page crawlable and put noindex on it, via meta tag or X-Robots-Tag header. The robot arrives, reads the prohibition and drops the document. If the page is deleted outright it should return 404 or 410. Blocking a page you want removed in robots.txt is the direct route to keeping it indexed — the robot can read neither your prohibition nor your status code.

Checklist: what to verify when pages are not indexed

• The page returns 200 on the live domain, not 404, 403 or 503.
• robots.txt does not block the address, and there is no Disallow: / left over from staging.
• The CSS and JS needed for rendering are not blocked from crawling.
• The HTML contains no <meta name="robots" content="noindex">.
• The response headers contain no X-Robots-Tag: noindex — verified on production after the latest release.
• rel="canonical" points at the address itself, not at the homepage or another page.
• There is no redirect chain: the address opens in one hop.
• The address is in sitemap.xml, and the sitemap holds only canonical pages returning 200.
• At least one internal link from a crawlable page points to the page.
• The content differs from neighbouring pages by more than a name.
• The server stays stable under load — according to monitoring data, not a single check.
• The address status and exclusion reason were read in a webmaster console, not inferred from a site: query.

The first six points close in a single pass: the HTTP header analyser shows the status code and X-Robots-Tag, the robots.txt checker resolves the rules for a specific address, and the SEO audit collects directives, canonical addresses and duplicates across the whole site at once. If the page still stays out of the index after that, the problem is not technical, and the remaining work is with content and demand.