What is X-Robots-Tag?

X-Robots-Tag is an optional component in HTTP response headers used to inform search engines how to crawl and index web pages and non-HTML files.

Definition and Function of X-Robots-Tag

X-Robots-Tag can be used for HTML and non-HTML files (such as PDFs, images, etc.), setting crawling and indexing instructions through HTTP response headers.

Common instructions include: noindex, nofollow, none, noarchive, nosnippet, etc.

HTTP/1.1 200 OK X-Robots-Tag: noindex, nofollow

Different instructions can be set for different crawlers, for example:
X-Robots-Tag: googlebot: noarchive, nofollow

Importance of X-Robots-Tag

X-Robots-Tag supports regular expressions, can batch control non-HTML file crawling, and is a more flexible SEO tool than meta robots.

  • Applicable to non-HTML files (such as PDFs, images)
  • Can be batch applied to entire directories or subdomains
  • Supports assigning different instructions to different crawlers

Common Instructions and Setup Methods

  • noindex: Do not display this page or file in search results
  • nofollow: Do not follow links on the page
  • none: Equivalent to noindex, nofollow
  • noarchive: Do not display cached pages
  • nosnippet: Do not display snippets or previews

Set X-Robots-Tag in server configuration files such as Apache, NGINX, etc., to achieve batch control.

Frequently Asked Questions

What's the difference between X-Robots-Tag and meta robots?

Meta robots only applies to HTML pages, while X-Robots-Tag can be used for all file types and is more flexible.

How to view X-Robots-Tag?

You can view X-Robots-Tag in HTTP response headers through browser developer tools or SEO toolbar.