Glossary · Glossary
Meta Robots
Meta robots directives tell search crawlers how to index or display a specific page.
Meta robots directives are page-level instructions that tell compliant search crawlers how to index or display a specific page. They can be placed in an HTML <meta> tag or, for non-HTML resources, in an HTTP header such as X-Robots-Tag.
These directives are useful when a page can be public but should not be indexed, followed, cached, translated, or displayed with certain snippet behavior. They are a precise page-level control, but only if crawlers can access the page and read the directive.
Why it matters
Meta robots directives protect publication boundaries. A site may need public preview pages, internal utility pages, duplicate feed pages, or temporary campaign pages that should not appear in search results. A clear directive can keep those URLs out of the index while allowing crawlers to verify the instruction.
For content production, the most important use case is separating published, indexable pages from pages that are live for review but not ready for search.
How it differs
Robots.txt controls crawler access before a page is fetched. Meta robots directives are read after the page is fetched. That means a page blocked by robots.txt may not expose its meta robots directive to the crawler.
Noindex is one specific meta robots directive. Meta robots is the broader mechanism that can include indexing, following, snippet, and preview instructions.
Examples
<!-- Ask compliant search engines not to index the page -->
<meta name="robots" content="noindex,follow" /><!-- Allow indexing but limit snippet length -->
<meta name="robots" content="max-snippet:160" />X-Robots-Tag: noindexThe HTTP header form is often used for PDFs, feeds, files, or other responses that do not have an HTML <head>.
How teams use it
Teams use meta robots directives for review pages, duplicate utility pages, expired pages that remain useful to users, or file responses that should not appear in search. A small checklist:
| Need | Common directive |
|---|---|
| Keep a page out of search results | noindex |
| Discourage link following from a page | nofollow |
| Limit displayed snippets | max-snippet |
| Control image or video previews | preview-related directives |
Common misunderstanding
Meta robots is not a secret publishing layer. If a page contains sensitive content, do not rely on noindex or meta robots. Remove the content from public access or protect it properly.
Read next
Use these glossary paths to move from the definition into adjacent concepts, topic clusters, and operator guides.