Hiding From Spiders

Most Webmasters spend their time trying to get their web pages into the search engine index as quickly as possible.

So why would you want to keep your web pages out of the index?

There are in fact some good reasons why you would want to hide some of your website from the search engine spiders. Among them:

• The content is private
• It is duplicate (Wordpress produces multiple URLs for the same content).
• You don’t want it to leak page rank. (contact us , privacy policy pages etc)
• It is not worth being indexed

So how do you do it… Find out in the Hiding from spiders article.