Robots.txt vs Noindex: Why Blocking Crawlers Does Not Remove Pages From Search
Two mechanisms control how search engines treat a page, they solve different problems, and confusing them is one of the most consistently damaging mistakes in technical SEO. The distinction moved from a specialist concern to a mainstream news story this week when Anthropic’s Claude conversation pages appeared in Google and Bing results despite a robots file instructing crawlers not to access them — the pages reportedly lacked a noindex directive.
The short version
A robots.txt disallow rule tells a crawler not to request a URL. A noindex directive tells a search engine not to include a URL in its index.
These are not two ways of saying the same thing. The first controls access. The second controls listing. A URL can be excluded from crawling and still appear in search results, which is exactly what makes the confusion dangerous.
Why a blocked page still gets indexed
Search engines learn about URLs from many sources beyond crawling the page itself. Links from other sites, links from other pages on the same site, sitemap submissions, browser telemetry, and third-party data all surface URLs independently.
When a search engine discovers a URL that robots.txt forbids it to fetch, it does not automatically discard the URL. It may index the address based on external signals — anchor text from inbound links, the URL structure itself, references elsewhere on the web — and display it in results with no description, sometimes with a note that no information is available for the page.
The result is the worst of both configurations: the page appears in search, and the site owner has no control over how it is described, because the engine was never permitted to read it.
The trap that catches most sites
The failure mode follows a predictable sequence. A site owner wants pages kept out of search. They add a disallow rule to robots.txt and also add a noindex tag to the pages. This appears thorough. It is broken.
The crawler obeys the disallow rule and never fetches the page. Because it never fetches the page, it never sees the noindex tag. The noindex directive is delivered in the page’s HTML head or in an HTTP response header, and both require the crawler to make the request.
The disallow rule prevents the noindex from ever being read.
The correct configuration
To remove a page from search results, allow crawling and serve a noindex directive. The crawler must be able to fetch the page in order to receive the instruction.
In HTML, place the directive in the head section as a meta robots tag with the value noindex. For non-HTML resources such as PDFs or images, use the X-Robots-Tag HTTP response header with the same value, since those file types have no head section to hold a meta tag.
Once the engine has processed the noindex and dropped the URL from its index, a disallow rule can be added afterward to save crawl budget. The order matters: index removal first, crawl blocking second. Reversing the order strands the page in the index permanently.
When robots.txt is the right tool
Robots.txt remains appropriate for controlling crawler load rather than search visibility. Typical valid uses include blocking infinite URL spaces generated by faceted navigation or calendar parameters, preventing crawlers from consuming resources on internal search result pages, restricting access to staging environments that should not receive crawl traffic, and reducing load on expensive dynamic endpoints.
In each case the goal is bandwidth and crawl efficiency, not index removal.
When neither tool is sufficient
Robots.txt is a public file at a predictable location and is advisory. Well-behaved crawlers honor it. Malicious scrapers read it as a directory of what a site owner considers sensitive.
Noindex removes a page from search results but does nothing to prevent direct access by anyone holding the URL.
Neither mechanism is a security control. Content that must not be accessible requires authentication, authorization, or removal from the public web. Anything that can be reached by an unauthenticated request over the open internet should be treated as potentially public regardless of what the robots file or meta tags say.
Verification
After configuring either mechanism, confirm the result rather than assuming it. Fetch the page as a crawler would and inspect the response headers and HTML head for the directive. Query the search engine for the specific URL to confirm whether it currently appears. Use the search engine’s own URL inspection tooling to verify how the page is being interpreted, since that reflects the engine’s actual state rather than the site’s intended configuration.
The gap between intended and actual configuration is where indexing incidents happen, and it is only visible by checking.