What are robot meta tags?

Robots meta directives (sometimes called "meta tags") are pieces of code that provide crawlers instructions for how lớn crawl or index website page nội dung. Whereas robots.txt tệp tin directives give bots suggestions for how lớn crawl a website"s pages, robots meta directives provide more firm instructions on how khổng lồ crawl and index a page"s content.

Bạn đang xem: Using noindex, nofollow html metatags: how to tell google not to index a page in search

There are two types of robots meta directives: those that are part of the HTML page (lượt thích the meta robotstag) and those that the web server sends as HTTP headers (such as x-robots-tag). The same parameters (i.e., the crawling or indexing instructions a meta tag provides, such as "noindex" and "nofollow" in the example above) can be used with both meta robots & the x-robots-tag; what differs is how those parameters are communicated lớn crawlers.


Code sample:

While the general tag is standard, you can also provide directives to lớn specific crawlers by replacing the "robots" with the name of a specific user-agent. For example, to target a directive specifically lớn Googlebot, you’d use the following code:

Want to use more than one directive on a page? As long as they’re targeted khổng lồ the same "robot" (user-agent), multiple directives can be included in one meta directive – just separate them by commas. Here’s an example:

This tag would tell robots not khổng lồ index any of the images on a page, follow any of the links, or show a snippet of the page when it appears on a SERPhường.

If you’re using different meta robots tag directives for different search user-agents, you’ll need to lớn use separate tags for each bot.


While the meta robots tag allows you to lớn control indexing behavior at the page màn chơi, the x-robots-tag can be included as part of the HTTPhường header to control indexing of a page as a whole, as well as very specific elements of a page.

While you can use the x-robots-tag to lớn exexinh đẹp all of the same indexation directives as meta robots, the x-robots-tag directive offers significantly more flexibility and functionality that the meta robots tag does not. Specifically, the x-robots permits the use of regular expressions, executing crawl directives on non-HTML files, và applying parameters at a global màn chơi.

Xem thêm: Cách Dịch Tiếng Anh Sang Tiếng Việt Trên Google Chrome Cực Kỳ Đơn Giản


To use the x-robots-tag, you’ll need khổng lồ have sầu access to lớn either your website’s header .php, .htaccess, or hệ thống access file. From there, add your specific VPS configuration’s x-robots-tag markup, including any parameters. This article provides some great examples of what x-robots-tag markup looks lượt thích if you’re using any of these three configurations.

Here are a few use cases for why you might employ the x-robots-tag:

Controlling the indexation of content not written in HTML (lượt thích flash or video)

Blocking indexation of a particular element of a page (like an image or video), but not of the entire page itself

Controlling indexation if you don’t have access lớn a page’s HTML (specifically, to the section) or if your site uses a global header that cannot be changed

Adding rules lớn whether or not a page should be indexed (ex. If a user has commented over đôi mươi times, index their profile page)

SEO best practices with robots meta directives

All meta directives (robots or otherwise) are discovered when a URL is crawled. This means that if a robots.txt tệp tin disallows the URL from crawling, any meta directive sầu on a page (either in the HTML or the HTTP header) will not be seen & will, effectively, be ignored.

In most cases, using a meta robots tag with parameters "noindex, follow" should be employed as a way khổng lồ to restrict crawling or indexation instead of using robots.txt file disallows.

It is important lớn note that malicious crawlers are likely to lớn completely ignore meta directives và as such, this protocol does not make a good security mechanism. If you have private information that you don’t want to lớn make publicly searchable, choose a more secure approach, such as password protection, to keep visitors from viewing confidential pages.

You vày not need to use both meta robots and the x-robots-tag on the same page – doing so would be redundant.

Keep learning

Put your skills lớn work Prolets you run crawls, retìm kiếm và report on từ khóa ranking, and traông chồng your site"s SEO performance, including its accessibility, over time. Try it >>