What Is X-Robots-Tag? When Should You Use It?

You cannot put a meta tag inside a PDF. There is no head section in a JPEG, an MP4, or an Excel file. Yet these files still get indexed by Google every day, sometimes ones you never meant to be searchable at all. X-Robots-Tag is the one control that reaches every file type your server sends out, HTML or not.

X-Robots-Tag is an HTTP response header that delivers the same indexing and serving directives as a meta robots tag, noindex, nofollow, noarchive, and more.

It applies these rules at the server level to any file type, including PDFs, images, and videos that have no HTML head to hold a meta tag at all.

We covered the page-level version of these directives in our meta robots tags guide, and the crawling layer beneath both in our robots.txt guide. X-Robots-Tag is the piece that handles everything those two cannot.

A Bouncer vs. a Sign Inside One Room

Basic X-Robots-Tag Syntax

HTTP/1.1 200 OK
Date: Tue, 25 May 2026 21:42:43 GMT
X-Robots-Tag: noindex

A minimal response telling every crawler not to index this URL

Blocking Non-HTML Files at the Server Level

According to Google Search Central's official robots meta tag documentation, this is the primary reason X-Robots-Tag exists: controlling indexing for file types that simply cannot host an HTML tag.

# Apache: block all PDFs from indexing
<Files ~ ".pdf$">
  Header set X-Robots-Tag "noindex, nofollow"
</Files>

# Apache: block image files specifically
<Files ~ ".(png|jpe?g|gif)$">
  Header set X-Robots-Tag "noindex"
</Files>

According to Search Engine Journal's guide to X-Robots-Tag, these two blocks handle the most common real-world use case

Targeting Specific Crawlers Within the Header

According to MDN's X-Robots-Tag reference, a user agent can be specified directly inside the header value, letting different bots receive different rules from the same response.

X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: BadBot: noindex, nofollow

Googlebot gets only nofollow here; a specified bot named BadBot gets both noindex and nofollow

What Happens When Multiple Rules Combine

According to MDN's reference referenced above, when several crawler-specific rules apply, search engines sum the negative directives rather than picking just one.

The Full Directive Reference

Which Robots Control Should I Use?

Select the resource type to see the right tool for the job.

Why This Header Matters More at Scale

According to Andres SEO Expert's technical definition guide, implementation at the server level allows high-scale, programmatic control across entire directories, rather than editing individual page templates one at a time.

According to Ferdy's 2026 explanation of this header, for standard HTML pages and posts, a meta robots tag through an SEO plugin remains the simpler default. This header specifically earns its keep for downloads, PDFs, and files an SEO plugin cannot reach.

Real-World Use Cases Worth Knowing

According to Nuwtonic's complete guide to mastering this header, expiring campaign pages, event materials, and licensed documents are common candidates for combining noarchive with unavailable_after, automatically removing outdated resources from results without manual cleanup.

According to Acromatico's AI visibility glossary entry, teams managing large document libraries or media-heavy sites benefit most, since a single server rule can govern thousands of files that would otherwise need individual review.

How to Check for an Existing X-Robots-Tag

According to Rank-O-Saur's guide to the X-Robots-Tag header, the header is invisible on the page itself, so checking requires opening browser DevTools' Network tab, selecting the file, and reading the Response Headers section.

# Or check directly from the command line
curl -I https://example.com/document.pdf

The -I flag returns headers only, showing any X-Robots-Tag present on that response

The One Trap Shared With Meta Robots Tags

According to Search Roost's guide to HTTP-level index controls, blocking a URL in robots.txt prevents the crawler from ever fetching it, which means any X-Robots-Tag on that response is never seen or followed either.

FAQs on X-Robots-Tag