How Google Crawls and Indexes Your Website
Before Google can rank any of your pages, it has to find them, render them, and decide they're worth keeping in its index — and each of those steps has failure modes that silently kill your rankings.
Rankings are downstream of indexing. If Google hasn't indexed a page, that page cannot rank — for anything, ever. And yet many businesses invest months in content and link-building without checking whether Google has even found the pages they're optimising. Understanding how crawling and indexing work is not just academic — it's the difference between an SEO investment that compounds and one that silently disappears into the void.
Google's process has three distinct phases: crawling (discovering URLs and fetching their HTML), rendering (executing JavaScript to see the page as a user would), and indexing (analysing and storing the page's content in Google's index). Problems at any phase have different symptoms and different fixes. Most small business websites have at least one indexing issue they're unaware of.
What is crawling and how does Googlebot discover your pages?
Googlebot discovers pages primarily through links — both external links from other websites and internal links within your own site. It starts from a known URL, fetches the HTML, extracts every link it finds, and adds those to its crawl queue. It also uses your XML sitemap as a direct map of which URLs exist and should be crawled. A page with no inbound links and not included in a sitemap may never be discovered.
XML sitemap: submit one via Google Search Console. It tells Google every URL you want crawled, with optional metadata about when each was last modified. Essential for new sites and large sites with deep URL structures.
Internal links: every page on your site should be reachable from at least one other indexed page. Orphaned pages — those with no internal links pointing to them — may never be crawled even if they're in your sitemap.
Robots.txt: a file at yourdomain.com/robots.txt that tells Googlebot which directories and URLs NOT to crawl. Misconfigured robots.txt files can accidentally block Googlebot from crawling entire sections of your site.
Crawl budget: for large sites, Google allocates a finite crawl budget — the number of URLs it will crawl per day. Slow server response times, redirect chains, and duplicate URLs consume crawl budget without producing indexed pages.
What is rendering and why does it matter for JavaScript sites?
After fetching a URL's HTML, Googlebot queues it for rendering — executing any JavaScript on the page to see the final DOM that users see. This is critical for sites built on React, Vue, Angular, or other JavaScript frameworks where content is generated client-side. If Googlebot can only see an empty HTML shell and the actual content is injected by JavaScript, it may fail to index that content at all, or may index a partial version of it.
Server-side rendering (SSR) is the most reliable approach for SEO: the server sends fully rendered HTML to both users and Googlebot before any JavaScript runs.
Static site generation (SSG) — where pages are pre-rendered at build time — is equally reliable for SEO.
Client-side rendering (CSR) only is the riskiest approach: Googlebot must execute JavaScript to see your content, and rendering is queued separately from crawling, potentially introducing days of delay between discovery and indexing.
Use Google Search Console's URL Inspection tool to check what Googlebot actually sees when it renders your pages. The "View Tested Page" function shows you the rendered HTML and a screenshot — if your content is missing there, Googlebot isn't seeing it either.
A page that looks perfect in a browser but renders as a blank shell to Googlebot is invisible to search. Rendering matters as much as content.
What determines whether Google indexes a page?
After rendering, Google decides whether to index a page based on several factors: the page's content quality (thin, duplicate, or low-value content may be crawled but not indexed), any indexing directives in the page's <head> tag (a noindex meta tag explicitly blocks indexing), the canonical tag (a rel="canonical" tag pointing to a different URL tells Google that URL is the preferred version to index), and duplicate content detection (Google typically indexes only one version of substantially identical pages).
Check for accidental noindex tags: a single <meta name="robots" content="noindex"> in a page's <head> will prevent that page from appearing in search results. This is one of the most common causes of "why isn't this page ranking at all?"
Canonical tags: ensure your canonical tags point to the intended URL and are self-referential on the page you want indexed. A canonical pointing to the wrong URL silently redirects Google's indexing signal.
Thin content: Google's Helpful Content guidelines mean pages with very little substantive content — single-paragraph pages, template pages with placeholder text, duplicate variation pages — may be crawled and not indexed at Google's discretion.
How do you check your indexing status?
Google Search Console is the primary tool. The Index Coverage (or Pages) report shows you which pages Google has indexed, which it has crawled but not indexed, and which it has discovered but not yet crawled — with specific reasons for each non-indexed page. Common non-indexed reasons include "Page with redirect" (a redirect chain), "Duplicate without user-selected canonical," "Crawled — currently not indexed" (Google crawled it but chose not to index it), and "Blocked by robots.txt." For a fast spot-check on any specific URL, use the site: search operator in Google: type site:yourdomain.com/your-page-slug and see if it appears in results. This is not exhaustive, but it confirms whether a specific URL is in the index. Understanding indexing connects directly to site architecture — how your site is structured determines how efficiently Google can crawl all of it.
How long does it take for a new page to get indexed?
For an established site with regular crawls, new pages submitted via sitemap or internal-linked from high-authority pages typically get indexed within a few days to two weeks. For new domains or sites Google rarely crawls, it can take weeks to months. The URL Inspection tool in Search Console has a "Request Indexing" function — using it after publishing a new page signals to Google that you want it crawled soon. It doesn't guarantee speed, but it does move the page to the front of the crawl queue.
Can Google index pages you don't want it to?
Yes, and this is a common problem. Admin pages, thank-you pages, duplicate filtered versions of product pages, and staging environments can all end up in Google's index. For admin pages and staging, block via robots.txt (preventing crawl) or noindex (preventing indexing, but still allowing crawl). For duplicate product filter URLs (like /shoes?color=red&size=12), use canonical tags to point to the clean category URL.
What happens if Google indexes a page and then de-indexes it later?
Google continuously re-evaluates indexed pages as it re-crawls them. A page can be de-indexed if it becomes thin (content is removed), if a noindex tag is added, if the page is deleted and returns a 404, or if Google determines the page is now low-quality relative to its standards. Sudden drops in indexed page count in Search Console are a red flag worth investigating immediately.
Keep reading
Crawling and indexing are the foundation — site architecture and URL structure covers how to make your site as easy to crawl as possible. And the technical SEO checklist includes crawl and indexing audits as part of a complete 15-item review. For context on how long it takes SEO to produce results, indexing speed is often the hidden bottleneck.
Sources
- Google Search Central — how Google crawls, renders, and indexes pages. developers.google.com/search
- Ahrefs — indexability guide, crawl budget optimisation, and JavaScript SEO. ahrefs.com/blog
- Search Engine Journal — Google rendering pipeline and JavaScript indexing research. searchenginejournal.com
Worried some of your pages aren't being indexed? Get a free Brand & Tech Assessment and we'll audit your Search Console coverage report and find every indexing gap.
Book a free Brand and Tech Assessment to see exactly how we would grow your organic visibility.
