Who is a robots META tag and what does it do?

Published by Christian | Filed under SEO

“Who is your Daddy and what does he do?” - Arnold Schwarzenegger in Kindergarten Cop

A robots META tag is an HTML tag you can use on any page on your website to tell search robots to either:

• Index the page (”INDEX”)
• Not index the page (”NOINDEX”)
• Follow the links on the page (”FOLLOW”)
• Not follow the links on the page (”NOFOLLOW”)

By default, if robots don’t see a robots meta tag, they’ll automatically assume that you want the content on the page indexed and all of the links (if not specifically marked as being nofollowed) followed.  So when you see a robots META tag display:

‘<META NAME=”ROBOTS” CONTENT=”INDEX, FOLLOW”>’

Either the webmaster doesn’t know that not having a robots META tag defaults these characteristics or the webmaster is making double-sure that robots know they want their page indexed and links followed.  In any case, this redundancy is not needed.

You have to be careful with this tag.  This single line of code can prevent your page from being indexed and all links on that page nofollowed.  Depending on what page a “NOINDEX, NOFOLLOW” robots META tag could be on, it could spell disaster for any kind of organic search success.  For example:

In this case, you have a robots META tag of NOINDEX and NOFOLLOW on your  Services page and there are no links pointing to either Service 1 or Service 2 pages internally or externally, these pages will not appear in any search engine result pages (SERPs).  The obvious times to use this tag is if you’re working on a section of your site of which there are no links pointing to those pages and you don’t want them to appear in search results.

If you’re hosting your own Wordpress blog, you should pay extra careful attention to see if there is a NOINDEX, NOFOLLOW META robots tag automatically included in your theme.  We’ve come across this situation where the blog author wasn’t aware of this and was writing content that had zero change of appearing in SERPs due to having this robots META tag included on every page.

So there you have it.  The robots META tag.  It can be a very useful tool if you want to tell robots to not index your content and not follow any links on the page.  Just be careful; reckless use of this tag can lead to your most content-juicy of pages not have any chance of appearing in search results.