Index

Index refers to the database of webpages a search engine has crawled, processed, and made eligible to appear in search results. A page that’s “indexed” can rank for queries; a page that isn’t can’t appear at all. The first prerequisite for any organic search visibility.

Different from “ranked” - being indexed means the page is in Google’s database; being ranked means it’s appearing for specific queries. A page can be indexed but ranking poorly, or indexed and ranking well, but it can’t rank without being indexed first.

How a page actually gets indexed

Three steps roughly in order:

Discovery. Search engines find the URL - through internal links, sitemaps, backlinks, or direct submission. Without discovery, the page might as well not exist.

Crawling. The crawler (Googlebot for Google) visits the URL and downloads the content. Modern Googlebot also renders JavaScript, so it sees JS-rendered content the way browsers do.

Indexing. The crawler processes the content and decides whether to add it to the index. Pages can be crawled but not indexed (because of low quality, duplicate content, noindex tag, or various other reasons).

What can prevent indexation

Five common blockers:

Noindex tag. Explicit instruction to search engines not to index. Useful for pages you don’t want in search (admin, internal search results, thin filter combinations).

Robots.txt blocking. Disallows crawling of certain URL patterns. Pages blocked here don’t get crawled, so they don’t get indexed.

Canonical pointing elsewhere. If page A has a canonical pointing to page B, Google indexes only B and consolidates signals there.

Quality threshold not met. Pages judged too thin, too duplicate, or too low-quality may be crawled but excluded from index. Most common silent blocker.

Discovery problem. Page exists but isn’t linked from anywhere Google has crawled. Doesn’t appear in any sitemap. Nobody knows it’s there.

How to check what’s indexed

Three methods:

Google Search Console Index Coverage report. Most authoritative source - Google itself telling you what’s indexed, what’s excluded, and (often) why.

site:yourdomain.com search. Quick approximation. Returns roughly the indexed pages but not always reliable for total counts.

URL Inspection tool in Search Console. Per-URL diagnostic showing indexation status, crawling history, and any blockers detected.

An example

A solo content creator’s site had 320 published articles. They expected most to be indexed. Search Console showed only 180 indexed.

The audit found three reasons for the gap. About 60 pages were caught by an overly broad noindex rule applied to “category” pages that had been mistakenly extended to article URLs. About 40 were duplicates of other articles (the team had published similar pieces over time without consolidating). About 40 were technically thin enough that Google had crawled and excluded them as low quality.

The fix took 6 weeks. Removed the bad noindex rule (recovered 60 pages). Consolidated 40 duplicate articles into stronger versions with 301 redirects. Substantially expanded 25 of the 40 thin articles; deleted the other 15. Final indexed count: 285. Organic traffic to recovered pages lifted by about 32% over the following quarter.

Related terms

• Google Algorithm - the system that determines indexation and ranking decisions
• Canonical URL - the directive that influences which version of duplicate content gets indexed
• Duplicate Content - a common cause of expected pages not being indexed
• Google Search Console - the tool for diagnosing indexation issues
• Error 404 - the status code that explicitly tells search engines a URL shouldn’t be indexed