For those of you unfamiliar with Robots.txt, it is a file that webmasters use to help control what aspects of their websites are indexed by search engines and what parts are not to be indexed. This is a great way to give you full control of how they view your website and help avoid duplicate content penalities, but using a Robots.txt has to be done properly and responsibly in order to avoid making matters worse or accidentally preventing your website from being indexed all together!

There are a lot of posts out there that talk about a Robots.txt file, how to set one up, etc. Unfortunately, many of these posts don’t apply specifically to WordPress and can often get confusing when you are trying to implement one for your website. Others tend to leave out a lot of important information that most people should have prior to attempting to add a Robots.txt file to their blog’s root directory.

Ask Apache is offering a Robots.txt file and updated Robots.txt file that you can download and use for your blog. You’ll of course want to customize it to meet your needs, including adding any directories that you have created since installing the base WordPress install that you don’t want them to be indexed.

I recommend you bookmark these posts if you plan to tackle adding a Robots.txt to your blog at some point in the future. Both posts provide a lot of valuable information and, as mentioned above, includes actual templates you can use for your website.

Kyle Eslick is WordPress enthusiast who took his passion for WordPress to the next level in 2007 by launching WPHacks.com as a place to share hacks, tutorials, etc. Follow Kyle on Twitter @KyleEslick!

  1. Ayush Saran says:

    We also added a disallow for feeds on our robots.txt for wordpress, because the feeds content duplicates the site content

    Disallow: */*/feed/*/
    Disallow: */feed