Noindex vs Nofollow vs Disallow: Which One Should You Actually Use?

Three directives. Three completely different jobs. And yet confusing them is one of the most common technical SEO mistakes that silently tanks sites.

This guide cuts straight to it: what each one does, what it doesn't do, and a decision table for every scenario you'll actually run into. If you want a deeper read on the broader crawl control landscape, start with the technical SEO guides hub.

The One-Line Difference

• noindex — crawl this page, but don't add it to the search index.
• nofollow — follow this link (or links on this page), but don't pass PageRank through it.
• disallow — don't crawl this URL at all.

That's it. Three tools, three layers of the crawl/index/link-equity pipeline. They don't overlap. They don't substitute for each other.

The Most Dangerous Misconception

Most people believe that putting a URL in robots.txt under Disallow keeps it out of Google's index.

It doesn't.

Disallow only tells Googlebot not to crawl the URL. If another page on the web links to that URL, Google can still discover it, list it in search results, and show the URL — just without a snippet. You'll see "A description for this result is not available because of this site's robots.txt."

If your goal is to remove a page from the index, you need noindex — and Google must be able to crawl the page to read that tag.

This leads directly to the most consequential mistake in robots.txt management:

What Each Directive Actually Controls

noindex

Delivered via:

• Meta tag: <meta name="robots" content="noindex">
• HTTP response header: X-Robots-Tag: noindex

What it does:

• Removes the page from Google's index (and Bing's, if you use name="robots")
• Googlebot still crawls the page — it needs to read the tag
• Links from that page still get crawled (unless you add nofollow)
• PageRank can still flow through links on noindexed pages

What it doesn't do:

• Prevent crawling (use Disallow for that)
• Remove the URL from third-party indexes or caches immediately
• Apply instantly — Google needs to recrawl before the page drops out

nofollow

Two forms:

1. Link-level: <a href="/page" rel="nofollow">text</a> — affects only that link
2. Page-level: <meta name="robots" content="nofollow"> — affects all links on the page

What it does:

• Tells Google not to follow the link for PageRank purposes
• The linked-to page can still be crawled via other paths
• Treated as a hint by Google (not a directive) since 2019 — Google may still follow nofollow links

What it doesn't do:

• Prevent the linked page from being indexed
• Stop Google from crawling the destination (it will if it finds the URL another way)
• Remove the link from appearing in the rendered HTML

disallow

Delivered via robots.txt:

User-agent: *
Disallow: /admin/

What it does:

• Tells compliant crawlers not to request that URL
• Saves crawl budget — pages aren't fetched at all
• Applies to the entire path prefix (unless you use $ for exact match)

What it doesn't do:

• Remove the page from the index
• Prevent Google from discovering the URL via links
• Apply to non-compliant bots (scrapers ignore robots.txt)

Decision Table: Which Directive to Use

*Paginated archives: Google now generally handles these fine. If you're seeing index bloat, noindex page 2+ is reasonable — but don't disallow, or Google can't read the noindex.

**Internal search results: disallow saves crawl budget; noindex ensures they don't rank. Using both is the one exception to the "never combine" rule — but only if you're absolutely certain you don't want them indexed and don't care if Google discovers them through links.

Real Scenarios

Scenario 1: E-commerce tag pages

Your WooCommerce store generates tag pages like /tag/blue-widget/ with three products on each. Thin content, no real search volume.

Use: noindex. Don't disallow — you want Google to crawl them so it reads the tag. Don't nofollow — the links to products on those pages should pass equity.

<meta name="robots" content="noindex, follow">

Scenario 2: Affiliate links

You're linking to Amazon with your affiliate ID. You don't want to pass PageRank through a commercial relationship.

Use: nofollow on the link. This is the canonical use case.

<a href="https://amazon.com/product?tag=yoursite-20" rel="nofollow sponsored">Buy on Amazon</a>

Use sponsored if you want to be precise about Google's link attribute taxonomy — it's a superset of nofollow for paid/affiliate links.

Scenario 3: Staging site

You're running staging.yoursite.com and don't want it indexed.

Use: disallow in robots.txt (applied to the entire staging domain). Optionally add a site-wide noindex header too — belt and suspenders, but don't rely only on noindex for staging since you probably don't want the crawl happening either.

User-agent: *
Disallow: /

Scenario 4: Internal search results

Your site has a search at /search?q=widget. These URLs are being crawled and occasionally appearing in Google.

Choose based on the goal. To remove an indexed search-result URL, let the crawler fetch a noindex directive in HTML or an HTTP header. A robots.txt block prevents that fetch, including reading an X-Robots-Tag header. Use disallow for crawl control when that is the intended policy; it does not guarantee removal from search.

But don't add <meta name="robots" content="noindex"> in the HTML and also block via robots.txt — the disallow blocks the crawl before the tag is ever read.

The Canonicalization Alternative

For equivalent duplicate representations, review canonicalization. Distinct pagination is different: later pages usually need unique URLs and self-canonicals. A canonical is a preference signal, not a guaranteed removal mechanism. See canonical diagnosis.

For faceted navigation, choose crawl control, indexability and canonical policies separately. A blocked URL prevents Google from reading its canonical and noindex directives. See faceted navigation SEO.

Quick Reference: Syntax

noindex meta tag:

<meta name="robots" content="noindex">
<meta name="robots" content="noindex, follow">
<meta name="robots" content="noindex, nofollow">

noindex via HTTP header (for PDFs, non-HTML files):

X-Robots-Tag: noindex

nofollow on a specific link:

<a href="/page" rel="nofollow">anchor text</a>

nofollow all links on a page:

<meta name="robots" content="nofollow">

disallow in robots.txt:

User-agent: *
Disallow: /path/          # blocks /path/ and all children
Disallow: /exact-page$    # blocks only this exact URL
Disallow: /*.pdf$         # blocks all PDF files

Summary

• noindex removes pages from the index. Google must crawl to read it.
• nofollow blocks PageRank through a link. Doesn't affect crawling or indexing.
• disallow stops crawling. Doesn't prevent indexing if the URL is discovered via links.
• Never put noindex on a page you've also disallowed — Google can't read what it can't fetch.
• When in doubt: can Google crawl it? Should it rank? Should links on it pass equity? Answer those three questions and the right directive picks itself.

For generating and testing your robots.txt directives before they go live, use the robots.txt generator and tester to validate syntax.