Now Reading: Meta Robots Tags Explained with HTML Code Examples
-
01
Meta Robots Tags Explained with HTML Code Examples
The meta robots tag is one line of HTML and one of the easiest ways to accidentally deindex your site. Real code for every directive, the #1 mistake that breaks noindex silently, and a live builder tool.
One misplaced meta robots tag has deindexed entire websites overnight. It is one line of HTML, and it is also one of the easiest ways to accidentally tell Google to forget your site exists. Here is exactly how the tag works, every directive it supports, and the real code for each one.
A meta robots tag is an HTML element placed in a page's head section that tells search engine crawlers whether to index the page and whether to follow its links.
Unlike robots.txt, which blocks crawling entirely, the meta robots tag lets a page be crawled but controls what happens after.
We covered a closely related file in our XML sitemap guide. Never include a page carrying a meta robots noindex tag in your sitemap, since the two signals directly contradict each other.
is the default behavior Google assumes when no meta robots tag is present at all
the year Google began treating nofollow as a hint, not a strict directive
rule above all others: Googlebot must crawl a page to ever see its meta robots tag
Basic Meta Robots Tag Syntax
Every meta robots tag follows the same structure, placed inside the page's head element.
<head> <meta name="robots" content="noindex, follow" /> </head>
The basic meta robots tag structure, placed in the head section of any page
The Four Core Directive Combinations
These four combinations cover the overwhelming majority of real-world use cases.
index, followDefault. No tag needed.index, nofollowShow in results, don't pass link equity.noindex, followHide page, still discover its links.noindex, nofollowFully hidden. No links passed at all.Noindex, Follow: Thank-You and Confirmation Pages
<meta name="robots" content="noindex, follow" />
Keeps a page out of search results while still letting Google discover linked pages
Noindex, Nofollow: Admin and Staging Pages
<meta name="robots" content="noindex, nofollow" />
Fully hides a page from indexing and prevents any link equity from passing through it
Targeting a Specific Crawler
According to MDN's official meta robots documentation, replacing the name attribute lets a directive apply to one specific bot rather than every crawler.
<meta name="googlebot" content="noindex" /> <meta name="bingbot" content="noindex" />
Applying noindex only to Google or Bing, leaving other crawlers unaffected
Additional Directives: Noarchive, Nosnippet, and Max-Snippet
| Directive | What It Does |
|---|---|
noarchive | Prevents Google from showing a cached version of the page |
nosnippet | Blocks any text snippet or video preview in results |
max-snippet:50 | Limits the text snippet shown to 50 characters |
max-image-preview:large | Controls the maximum size of an image preview shown |
unavailable_after | Automatically removes a page from results after a set date |
<meta name="robots" content="noarchive, max-snippet:50" /> <meta name="robots" content="unavailable_after: 2026-12-31T23:59:59+00:00" />
Combining multiple directives in one tag, and setting an automatic expiration date
X-Robots-Tag: Controlling Non-HTML Files
According to Google Search Central's official robots meta tag documentation, the X-Robots-Tag HTTP header carries the same directives but applies to files where an HTML head tag is not possible, like PDFs or images.
# Apache .htaccess example <FilesMatch "\.pdf$"> Header set X-Robots-Tag "noindex, nofollow" </FilesMatch> # NGINX example location ~* \.pdf$ { add_header X-Robots-Tag "noindex"; }
Applying noindex to every PDF file on a site through server configuration, not HTML
The One Mistake That Breaks Everything
This single error causes more noindex failures than any other mistake combined. Tap through both scenarios.
According to DigitalApplied's complete SEO reference, blocking a URL in robots.txt while also adding a meta robots noindex tag on that same page fails silently.
Googlebot cannot fetch the page, so it never sees the noindex directive at all, and the URL can still appear indexed with no description.
Allow crawling in robots.txt, then apply the noindex meta tag on the page itself. This lets Googlebot actually fetch the page, read the directive, and remove it from the index correctly.
Meta Robots Directive Builder
Select your options below to generate the exact meta robots tag for your situation.
Meta Robots Directive Builder
Builds a copy-paste ready meta tag based on your selections
Common Meta Robots Mistakes
Blocking Crawling and Adding Noindex Together
As covered above, this is the single most common failure. The noindex tag never gets seen if robots.txt already blocks the page.
Injecting the Tag Only Through JavaScript
According to Devender Gupta's guide to meta robots tags, Googlebot processes the raw HTML first, then the rendered DOM after JavaScript executes, creating a delay before a JS-injected directive takes effect.
Conflicting Signals Across Plugins and Templates
According to User Growth's comprehensive guide to robots meta tags, if one plugin outputs index and a hardcoded tag says noindex, Google follows the most restrictive instruction, which is noindex, but the conflict itself signals a messier setup worth cleaning up.
Meta Robots Tag Placement Rules Worth Remembering
According to Conductor's meta robots tag guide, the tag should be placed once per page, inside the head section, and never referenced from a page that also carries an hreflang tag pointing to it, since hreflang signals a page should be indexed.
According to iRankly's guide to meta robots directives, avoiding noindex on paginated pages is another common recommendation, since search engines generally understand pagination well enough not to treat it as duplicate content requiring exclusion.
Meta Robots Tag Behavior Over Time
According to Search Roost's guide to meta robots directives, Google continues crawling noindex pages periodically to check whether the directive has changed, rather than abandoning them permanently after the first noindex is detected.
Over an extended period, a persistently noindexed page's crawl frequency drops, and its outbound links effectively behave as nofollow in practice, even without an explicit nofollow directive ever being added.
FAQs on Meta Robots Tags
> what_we_learn_today.log
Meta robots tags require crawling first; robots.txt blocks crawling entirely
Default behavior with no tag present is always index, follow
Never combine robots.txt disallow with a noindex tag on the same page
X-Robots-Tag handles PDFs, images, and other non-HTML files
Nofollow has been a hint, not a strict rule, since 2019
The most restrictive directive always wins when signals conflict
Related Posts
Previous Post
Next Post
Recent Posts
- Google Releases August 2026 Spam Update: What SEOs Need to Know
- YouTube Partner Program 2027 Updates: What Every Creator Needs to Know
- WebP vs AVIF: Which Image Format Is Better for SEO in 2026?
- Preload vs Prefetch vs Preconnect: Which Resource Hint Should You Use?
- Meta Robots Tags Explained: Index, Noindex, Follow, Nofollow, and More
