Short answer. When a site is missing from search results, check indexing before anything else: run a site: query, then open the Page indexing report in Google Search Console. The most common causes are a stray Disallow in robots.txt, a noindex tag or header, server errors during crawling, and simply a very new domain. This checklist walks you to the exact cause.
"Not in search" means three different problems
Before fixing anything, pin down which problem you actually have:
- The site is not indexed at all. A site: query returns zero results. Search engines either never discovered the site or were told to stay away.
- The site is indexed but does not rank even for its own brand name. Pages exist in the index, yet a search for your company name shows competitors and directories. Usually a quality issue, duplicate-domain confusion, or a weak homepage.
- The site is indexed but buried deep in the results. That is a ranking problem, not an indexing one — start with a full SEO audit and the DIY audit checklist.
This article covers the first two cases: pages missing from the index, or a site invisible even for branded queries.
How to check whether your site is indexed
The site: operator
Type site:yourdomain.com into Google. Zero results means the site is not in the index. If results appear, compare their number with the real page count: a large gap means whole sections have dropped out.
Google Search Console
The Page indexing report (formerly Coverage) splits URLs into indexed and not indexed, with a stated reason for every excluded group. The URL Inspection tool at the top of the console gives a verdict on a single page: whether it is indexed, which canonical Google selected, and when it was last crawled. The "Request indexing" button lives in the same tool.
Yandex specifics
If part of your audience comes from Yandex, add the site to Yandex.Webmaster. The "Pages in search" report lists indexed and excluded URLs with reasons, "Crawl statistics" shows which response codes the robot receives, and the url: operator checks a single page directly in Yandex search.
Technical reasons a site blocks itself
When a site is not indexed, the cause is technical far more often than mysterious. Work through these in order.
robots.txt disallows crawling
The classic case: the site was built on a staging domain closed to robots, and the config travelled to production untouched. This file blocks the entire site:
User-agent: * Disallow: /
A healthy version opens the site, blocks only service sections, and points to the sitemap:
User-agent: * Disallow: /admin/ Disallow: /search/ Sitemap: https://example.com/sitemap.xml
The single most common cause of a "vanished" new site is a forgotten Disallow: / left over from staging. Check robots.txt first, always — it is one line that switches the whole site off in search.
noindex meta tag and X-Robots-Tag header
The ban can hide in the page HTML (<meta name="robots" content="noindex">) or in an HTTP header that never appears in the source at all. One command reveals the response headers:
curl -I https://example.com/ HTTP/2 200 content-type: text/html; charset=utf-8 x-robots-tag: noindex, nofollow
An x-robots-tag: noindex line means the server forbids indexing at the response level — CMS plugins and web-server configs set it without the owner noticing.
Canonical points to another page
If rel="canonical" on every page links to the homepage — a typical template bug — Google treats the whole site as duplicates of one URL and drops the rest from the index.
Pages return 4xx, 5xx, or redirects
Only pages that answer 200 get indexed. URLs returning 404, 403, 500, or long redirect chains fall out of the index. A broken-link check finds such URLs in bulk.
Password protection and anti-bot filtering
HTTP auth across the site, a captcha on first visit, or a firewall that filters crawlers by User-Agent or IP all look like "site unavailable" to a search robot. Use URL Inspection to see the page exactly as Googlebot does.
Missing sitemap.xml
Without a sitemap, the crawler discovers pages only by following links — slow for large or brand-new sites. An up-to-date sitemap.xml surfaces sections the robot would otherwise take weeks to reach.
JavaScript rendering without SSR
If content appears only after JavaScript executes, the crawler may see an empty page. Google renders JS with a delay and not always completely; other engines are stricter. For SPAs, use server-side rendering or prerendering for the pages that must rank.
New sites: how long indexing takes and how to speed it up
A fresh domain with no inbound links gets indexed slowly: think days to weeks, not hours. That alone is not a problem. To speed things up:
- add the site to Google Search Console and submit sitemap.xml;
- run key pages through URL Inspection and hit "Request indexing";
- set up IndexNow to notify supporting search engines about new and changed URLs instantly;
- earn the first external links — social profiles, directories, industry sites — crawlers discover you through them;
- in Yandex.Webmaster, use "Recrawl pages" for the same purpose.
Content quality and penalties
If the technical layer is clean but pages keep dropping out, look at the content. Search engines decline to index pages they classify as thin, duplicated, or generated without editing — the "Crawled — currently not indexed" and "Discovered — currently not indexed" groups in Search Console usually point exactly there.
A separate scenario is a hacked site: injected doorway pages or malicious code get a site flagged and mass-removed from results. Scan it with a malware checker and walk through the website security checklist. While you are at it, make sure your structured data validates: broken markup hurts how results display.
Crawl availability: downtime silently removes pages
Crawlers index only what they can reach. If the server times out or returns 5xx at the moment of a crawl, the robot lowers its crawl rate and gradually drops unreachable URLs from the index. Slow responses make it worse — test your page speed and set up uptime monitoring so that you learn about outages before the crawler does. The same applies to region-level blocking: if the site is unreachable from the country where a search engine's crawlers operate, its index there quietly erodes.
A crawler will not wait for your site to come back up. A page that answers with errors a few times in a row falls out of the index and takes weeks to return. Uptime monitoring is an SEO tool, not just an ops tool.
Step-by-step diagnostic plan
| Symptom | Where to look | Likely cause | What to do |
|---|---|---|---|
| site: returns zero pages | robots.txt, meta tags | Disallow: / or noindex | Open the site to crawlers, request indexing |
| Page excluded as duplicate | Page indexing report | Canonical points to another URL | Set a self-referencing canonical |
| "Discovered — currently not indexed" | Page indexing report | Thin content or crawl budget | Improve the page, request indexing |
| Crawler gets error responses | Crawl stats, server logs | 5xx, timeouts, anti-bot filtering | Stabilize the server, allow crawlers |
| x-robots-tag: noindex in response | curl -I output | CMS plugin or server config | Remove the header, request recrawl |
| Pages drop out in waves | Excluded pages reports | Thin content, duplicates, hack | Improve content, scan for malware |
Still no verdict? Go down the checklist:
- run site: — is the site in the index at all;
- open robots.txt and look for Disallow: / or blocked key sections;
- check the source and response headers (curl -I) for noindex and X-Robots-Tag;
- verify canonicals on key pages;
- make sure pages answer 200 rather than errors or redirect chains;
- submit sitemap.xml in Search Console;
- read the exclusion reasons in the Page indexing report — they name the cause directly;
- after fixing, request indexing for the affected URLs.
To cover the whole technical layer in one pass, run a full SEO audit — it checks robots.txt, meta tags, response codes, and indexing directives together.
Frequently asked questions
How long does indexing a new website take?
Anywhere from days to weeks, depending on how fast crawlers discover the site. Search Console, a sitemap, and indexing requests shorten the wait considerably compared with passive waiting.
Why is the site in Google but not in Yandex (or vice versa)?
The indexes are independent: different crawl speed, different quality thresholds, different JavaScript rendering. First check for bot-specific rules (for example, a Disallow aimed at one crawler), then open the webmaster panel of the engine where the site is missing — the exclusion reason is usually stated in the report.
Does moving to HTTPS affect indexing?
Yes: every URL changes, and search engines re-map the site to the new protocol. With correct 301 redirects the dip is temporary. Details in our article on how HTTPS affects SEO.
A page was removed from the index — how do I get it back?
Find the exclusion reason in Search Console, fix it — remove noindex, restore a 200 response, deduplicate the content — then request indexing for the URL. Recovery starts within days, not instantly.
How do I check indexing for many pages at once?
Export the URL list from your sitemap and compare it with the Page indexing report export. For ongoing control, watch the indexed-pages trend weekly: a sharp drop is a signal to check the server and robots.txt before anything else.