Knowledge hub · Reference
Robots meta tags, X-Robots-Tag and robots.txt rules
Verified against official sources · Last checked 6 October 2026
Short answer
Robots directives tell search engines what they may crawl, index and show. robots.txt controls crawling; the robots meta tag and X-Robots-Tag header control indexing and snippets; the data-nosnippet attribute hides parts of a page from snippets. A blocked page cannot be read, so a noindex on it is never seen.
Which robots meta and X-Robots-Tag values exist?
| Directive | Effect | Supported by |
|---|---|---|
| noindex | Keeps the page or file out of search results; the crawler must be able to fetch the URL to see it. | |
| nofollow | Tells the crawler not to follow the links on the page. | |
| none | Shorthand for noindex plus nofollow. | |
| noarchive | Google no longer uses it because its cached-page link was retired; Bing treats it as a block on using the content in Copilot/Bing chat answers and for training its AI models. | Bing (Google: no longer used) |
| nocache | Bing only: content may still appear in chat answers but only as URL, title and snippet, and only those may be used for AI model training. Google ignores it. | Bing |
| nosnippet | Stops a text snippet or video preview from being shown; at Google it also keeps the content from being used directly in AI Overviews and AI Mode. | Google, Bing |
| max-snippet:[number] | Caps the snippet at the given number of characters (0 means no snippet, -1 lets the engine decide); Google applies the limit to AI Overviews and AI Mode too. | Google, Bing |
| max-image-preview:[none|standard|large] | Sets the largest image preview that may be shown for the page. | Google, Bing |
| max-video-preview:[number] | Caps video previews at the given number of seconds (0 means a still image only, -1 means no limit). | Google, Bing |
| notranslate | Asks Google not to offer a translated version of the page in results. | |
| noimageindex | Keeps the images on the page from being indexed. | |
| unavailable_after:[date/time] | Removes the page from results after the given date and time, given in a widely used format such as ISO 8601. | |
| indexifembedded | Lets content be indexed when it is embedded in another page through an iframe, even though it carries noindex; only works together with noindex. |
Which HTML attributes control snippets?
| Directive | Effect | Supported by |
|---|---|---|
| data-nosnippet | Excludes the marked span, div or section from snippets; Bing (since October 2025) also excludes it from Copilot and AI answers while the page stays indexed. | Google, Bing |
Which robots.txt rules exist?
| Directive | Effect | Supported by |
|---|---|---|
| Disallow | Blocks the named user agent from crawling matching paths; a blocked URL can still be indexed from links without its content. | Google, Bing |
| Allow | Re-opens a path inside a disallowed section for crawling; the most specific rule wins at Google. | |
| Sitemap | Gives the absolute URL of a sitemap or sitemap index file. | |
| Crawl-delay | Google ignores this field. Bingbot honours it as a 1-30 second window in which it fetches at most one page. | Bing (ignored by Google) |
Sources
- developers.google.com/search/docs/crawling-indexing/robots-meta-tag
- developers.google.com/search/docs/crawling-indexing/robots/robots_txt
- blogs.bing.com/webmaster/april-2020/Announcing-new-options-for-webmasters-to-control-their-snippets-at-Bing
- blogs.bing.com/webmaster/september-2023/Announcing-new-options-for-webmasters-to-control-usage-of-their-content-in-Bing-Chat
- blogs.bing.com/webmaster/October-2025/Bing-Introduces-Support-for-the-data-nosnippet-HTML-Attribute
- blogs.bing.com/webmaster/May-2012/To-crawl-or-not-to-crawl,-that-is-BingBot-s-questi/