Meta Robots
HTML tag or HTTP header that tells search engines whether to index a page and follow its links. Common directives: noindex (don't index), nofollow (don't follow links).
Why It Matters
Meta robots gives you granular control over how search engines handle individual pages. Unlike robots.txt (which controls crawling), meta robots controls indexing. You can tell Google to crawl a page but not index it, or to index it but not follow its links.
This matters when you have pages that need to exist for users but shouldn't appear in search results - thank you pages, internal search results, filtered views, staging environments. Without proper meta robots directives, these pages dilute your site's quality signals.
In Practice
Use noindex for pages you don't want in search results: thin pages, internal search results, paginated archives (beyond page 1), and admin or staging pages. Use nofollow sparingly - typically only for links to untrusted content.
You can set meta robots in two ways: an HTML meta tag in the page head, or an X-Robots-Tag HTTP header. The HTTP header is useful for non-HTML files (PDFs, images) where you can't add HTML tags.
Audit your noindex tags regularly. Accidentally noindexing important pages is one of the most common and devastating technical SEO mistakes.
Common Mistakes
Noindexing pages that should be indexed (often during a staging-to-production migration). Using noindex AND blocking in robots.txt simultaneously - Google can't see the noindex tag if it can't crawl the page. Confusing noindex (controls indexing) with robots.txt disallow (controls crawling).
Related Terms
Glossary
Robots.txt
A file telling search engine crawlers which parts of your site they can access.
Glossary
Indexing
Adding a crawled page to Google's database so it can appear in search results.
Glossary
Canonical Tag
Tells search engines which version of a page is the original when duplicates exist.
Glossary
Technical SEO
Optimising your site's infrastructure so search engines can crawl, render, and index it.
Know the Words.
Now See Them in Action.
Free teardown. No jargon. Just what's broken and how to fix it.
Get The Teardown