In this article we’ll cover META Robots Tag and why they are important for you to understand and their implications to your website with a practical example from my own experience of using a Wordpress theme template.

Within standard HTML code there are tags for the Title of your page which tells the spiders used by search engines what your website is all about, but its very limited information. This is where Meta Tags come in, as they enable you to tell everyone exactly what your site is about and allow search engines to index your site and pages into their databases. There are Meta Tags for Author, Description and Keywords which help you more carefully direct search engines on what you think is important about your site, and we will cover these in more detail in a separate tutorial, because they are quite important to understand.

What are META Robots Tags? These are specific instructions to the search engine spiders (robots) that crawl the internet collecting information on pages for indexing. To tell you the truth, I’d seen this tag added in some of my wordpress themes and paid no attention to it. How I came upon this was when I ran my SEO report and it came up with the following errors or things that needed attention:

META Robots

1. You have the ‘NOINDEX‘ value set in your META Robots tag! This value disallows any search engine to read your site, meaning that this page will never appear in search results. If you unintentionally put ‘NOINDEX’ into your META Robots, remove it with your favorite HTML editor.

2. You have the ‘NOFOLLOW‘ value set in your META Robots! This value disallows search engines to read pages linked from this page. If you accidentally put ‘NOFOLLOW’ into your META Robots, remove it with your favorite HTML editor.

I had seen the tag before, but it looked harmless enough until I saw this SEO report. NOW PLEASE READ THE NEXT BIT CAREFULLY, BECAUSE IT IS VERY IMPORTANT. The implications of this are basically that your website does NOT want search engines to read your site, which means your pages won’t be indexed by Google and the other search engines so how on earth is anyone going to find you!!!! Why anyone would want to do this is beyond me, but I guess if you’re a corporate business or an institution you may have a good reason for keeping things private. BUT, and this is an IMPORTANT but, why put it in a Wordpress Template!!! Anyone using Wordpress is using it to build a web presence through a blog site and the most important thing these sites is to get readership, and how can you get that readership without allowing search engines to index your pages. I’m not sure whether this was put in deliberately, or whether search engines ignore this statement, but my SEO Report recommended their removal, and I’m going to show you what I did and how you can check this yourself and avoid falling into the trap.

Steps to Find and Remove Harmful META Robots Tags NOINDEX and NOFOLLOW

1. The first thing I did was to log into my Wordpress Admin dashboard and had a closer look at all my template files. I checked my header, the main index, and pages file and I could n’t find the statement anywhere, so I was a little puzzled as to where those META Robots tags NOINDEX and NOFOLLOW came from.

2. I went back to my browser window, opened up my website and did View | Page Source, this will show you the actual HTML code that your web site is actually using to display the page on your website.

3. Next go to Edit | Find and type in the word “Meta” and click Find Next, this will highlight the meta-tags for , and you can search through this one at a time to uncover any META Robots tags NOINDEX and NOFOLLOW.

The beauty of this is that you can do this search at anytime, whether you have any SEO software or not, check for these tags and remove them before they cause you problems. So I’d now found my META Robots tag culprits, and knew they were there on my website in the HTML code, but the template did n’t show any sign of them. The problem is that the template files use a lot of PHP functions to make the task of displaying repetitive information easier, and the trick was to find the function that was causing this problem to occur. I could see from my source code, that the HTML code for these tags was being inserted right after the:

<link rel=”….”/> statements

So I went back to my Wordpress Header template file and looked at what was right after these statements. And there was the culprit, the last statement before the end of the ‘header’ code contained the following function:

<?php wp_head(); ?>

4. Remove the ‘php wp_head()‘ statement from your header template file, save it.

IMPORTANT Note: This is a temporary solution as I have discovered that one of my plug-ins (post_views) has stopped working, so I’m still looking into it to find a solution.  This particular function ‘php wp_head()’ always appears at the end of the ‘header’ section of the page, and is used by plug-ins and widgets to insert code during compilation.  So it would appear that one of my widgets is causing the problem, rather than the theme itself!!!  To read the SOLUTION to this problem please read my next article on this subject entitled How to Remove Meta Robots Tags in Wordpress

NOTE: There is no need to remove the PHP function below, please read this instead to solve this problem How to Remove Meta Robots Tags in Wordpress

5. Refresh your browser page for your web-site (just in case it is cached, you might need to re-open a new browser window) and take another look at the page source and see if the META Robots tags statements have now gone from the html source code.

From the warning message from my SEO report it was flagged as quite a critical thing to resolve, and I’m glad I did remove it soon before submitting my site to the search engines for indexing, but I wonder how many people out there who have used Wordpress theme templates have had these META Robots tags for NOINDEX and NOFOLLOW added to their website pages inadvertently, and have found a lack of visitors.

As a check, I looked through another Wordpress Theme template I had been using, and although I have n’t run the SEO report on it, looking through the HTML page source of the website there were no META Robots tags in this particular theme. So these tags are NOT there by default in all themes, so the warning only applies to themes that insert this META Robots tags. Regardless, it would do no harm to check the source code of your web pages to make sure it’s not there, as in a lot of cases HTML code is cut and pasted from one place to another.

As I looked into this further, I realised there was a positive action that you could take using this META Robots tags, and that was to use it to advise the search engines to INDEX your pages and to FOLLOW the links on your web pages. So what I have done now is not just remove the offending function in my header file, I have replaced it with two additional lines of HTML code as follows:

<META name=”robots” content=”index,follow”>

This will hopefully mean when my website is crawled the next time by the search engine spiders, they will not only INDEX my website, but FOLLOW all the links to build up a we that connects my website to the rest of the World Wide Web.

Since writing the above just now, I have thought about a good reason for not indexing pages and this is when you’re working on something during development, but these Meta Robots tags should be removed once a production release of a template is issued for general distribution.

This is all pretty new to me and as you can see I’m discovering things as we go along and I hope this article has been useful to you and may help eliminate any problems you may encounter with the inadvertent insertion of these META Robots tag.

Daniel Hunt

[Postscript: Please read the follow-up article on this subject before implementing any changes to your Header files, this will tell you what the full solution to this problem is: How to Remove Meta Robots Tags in Wordpress

........................................................................................................................................................

Previous / Next Articles

« « Blogging for Beginners using Blogger.com

How to Remove Meta Robots Tags in Wordpress » »

........................................................................................................................................................

technicalarticles.co.uk features regular articles on my blogging journey of creating a unique web presence and sharing with my readers ideas and tips for helping them do the same. Keep up to date with the latest posts from Daniel Hunt - Join the Community by Subscribing.