How To: Creating a Robots.txt for a WordPress Blog

Kyle Eslick — Tue, 26 Feb 2008 11:00:24 +0000

For those of you unfamiliar with Robots.txt, it is a file that webmasters use to help control what aspects of their websites are indexed by search engines and what parts are not to be indexed. This is a great way to give you full control of how they view your website and help avoid duplicate content penalities, but using a Robots.txt has to be done properly and responsibly in order to avoid making matters worse or accidentally preventing your website from being indexed all together!

There are a lot of posts out there that talk about a Robots.txt file, how to set one up, etc. Unfortunately, many of these posts don’t apply specifically to WordPress and can often get confusing when you are trying to implement one for your website. Others tend to leave out a lot of important information that most people should have prior to attempting to add a Robots.txt file to their blog’s root directory.

Ask Apache is offering a Robots.txt file and updated Robots.txt file that you can download and use for your blog. You’ll of course want to customize it to meet your needs, including adding any directories that you have created since installing the base WordPress install that you don’t want them to be indexed.

I recommend you bookmark these posts if you plan to tackle adding a Robots.txt to your blog at some point in the future. Both posts provide a lot of valuable information and, as mentioned above, includes actual templates you can use for your website.

Enjoy writing about WordPress? Get your blog more exposure by joining the WordPress Hacks writing team!

WordPress Hacks » Robots.txt

How To: Creating a Robots.txt for a WordPress Blog