Two mechanisms control how search engines treat a page, they solve different problems, and confusing them is one of the most consistently damaging mistakes in technical SEO. The distinction moved from a specialist concern to a mainstream news story this week when Anthropic’s Claude conversation pages appeared in Google and Bing results despite a robots file instructing crawlers not to access them — the pages reportedly lacked a noindex directive.