X-Robots-Tag Checker

An X-Robots-Tag checker reads the HTTP response headers of a URL and reports any X-Robots-Tag directives. Headers are how you control indexing of PDFs and other files that have no HTML head.

This tool fetches the address you enter once, from our server, and reports what a crawler would receive. The URL is not stored; only a short-lived rate-limit counter is kept. Results show the page at the moment of the check and can differ from what a logged-in browser or a bot with different headers sees.

How to read the result and act on it

Look at the directives returned in the header: noindex, nofollow, noarchive, nosnippet, max-snippet and similar. If the URL is a PDF, image or other file you want in search results, there should be no noindex. If it is a file you want kept out, such as a private download, noindex is the right control. When a directive carries a user agent prefix like googlebot: noindex, it applies only to that crawler, so check the scope matches your intent. If the header appears unexpectedly, look at server config, a CDN rule, a security plugin or an application layer that adds headers. Change it at the source and retest.

Common mistakes to avoid

The biggest trap is a server-wide rule that adds noindex to a file type or directory and catches pages that should rank. Because headers are invisible in a normal browser view, these go unnoticed for months. Another mistake is setting a header on a URL that robots.txt blocks, so crawlers never fetch the response and never see the directive. Be careful with CDN or proxy layers that add headers different from your origin. Also check that the header and any meta tag agree, because conflicting values leave the stricter one in effect. A result with no header means no directive was sent, not that the page is indexed.

When should I use X-Robots-Tag instead of a meta tag?

For non-HTML files such as PDFs and images, or when you need to apply rules at server level.

Can headers and meta tags conflict?

Yes. The most restrictive directive wins, so check both when a page fails to index.

How do I noindex a PDF file?

Send an X-Robots-Tag: noindex header with the PDF response, using your server config such as an Apache FilesMatch rule or an Nginx location block. A meta tag cannot be used because PDFs have no HTML head.

Can I see the X-Robots-Tag in Chrome DevTools?

Yes. Open DevTools, go to the Network tab, reload the page, select the main document and read the Response Headers section. Look for X-Robots-Tag in the list. A header checker shows the same without opening DevTools.

• Noindex Checker: Check whether a URL is blocked from indexing by a meta robots tag or X-Robots-Tag header.
• HTTP Header Checker: View the response headers of a URL: status, server, cache, content type and more.
• Robots Meta Tag Generator: Build robots meta tags and the matching X-Robots-Tag header.

Browse all 41 free technical seo tools, or see the full free SEO toolbox. If you would rather have this handled for you, take a look at our technical SEO services and the results from real projects.