Robots.txt vs noindex: which one actually blocks Google?

The short answer

robots.txt controls whether Googlebot visits a page. noindex controls whether Google listsit in search results. They operate at entirely different stages of Google's pipeline, and confusing the two is one of the most expensive technical SEO mistakes you can make.

How robots.txt works

The robots.txt file sits at the root of your domain (https://example.com/robots.txt) and tells crawlers which paths they are and are not allowed to fetch. A Disallow: /admin/ rule means Googlebot will not request any URL under /admin/.

The key word is request. Robots.txt is a crawling directive, not an indexing one. Google sees the rule before it ever loads the page content.

Note

Robots.txt is advisory, not mandatory. Most legitimate crawlers respect it. Bad actors ignore it entirely.

How noindex works

A noindex directive tells Google: "You can crawl this page, but do not include it in the search index." It can appear in two places:

As a meta tag: <meta name="robots" content="noindex">
As an HTTP response header: X-Robots-Tag: noindex

For Google to see and respect a noindex directive, it must first crawl the page. This is where most misconceptions arise.

The critical difference: crawling vs. indexing

Google's pipeline has two distinct stages:

Crawling — Googlebot fetches the page HTML
Indexing — Google processes and stores the page in its index

robots.txt intervenes at stage 1. noindex intervenes at stage 2. A page blocked by robots.txt never reaches stage 2. A noindex page gets crawled (stage 1) but not indexed (stage 2).

The worst mistake: combining both

Here is the scenario that silently kills indexing strategies: a developer adds a noindex meta tag to a page to prevent it from appearing in search, then also adds a Disallow rule in robots.txt for the same path.

Warning

If a page is blocked by robots.txt, Googlebot cannot read its noindex tag — because it never loads the page. Google may still discover the URL from external links and list it without a title or description snippet, because it cannot read the page content.

The result is a page that appears in search results with no description ("A description for this result is not available because of this site's robots.txt") — the opposite of what you wanted.

When to use robots.txt

Staging and development environments you want to block entirely
Internal admin dashboards behind authentication (belt-and-suspenders alongside auth middleware)
Large sets of thin or duplicate parameter-based URLs to preserve crawl budget
Assets (CSS, JS) that do not need individual indexing consideration

When to use noindex

Thank-you pages and confirmation pages after form submissions
Paginated archive pages beyond page 2–3
Faceted navigation URLs (e.g., /products?color=red&size=M)
Thin content pages you want Googlebot to crawl but not rank
Duplicate content that you want Google to understand but exclude from search

How to audit your configuration

Run these checks regularly:

Fetch your robots.txt and compare Disallow rules against your noindex pages — look for overlaps
Use Google Search Console's "URL Inspection" tool to see how Google sees any individual page
In GSC's Coverage report, check for pages listed as "Excluded by robots.txt" — if any should be noindexed instead, fix the configuration
Use a site crawler to detect pages with both a Disallow rule and a noindex tag