How To: Creating a Robots.txt for a WordPress Blog

For those of you unfamiliar with Robots.txt, it is a file that webmasters use to help control what aspects of their websites are indexed by search engines and what parts are not to be indexed. This is a great way to give you full control of how they view your website and help avoid duplicate content penalities, but using a Robots.txt has to be done properly and responsibly in order to avoid making matters worse or accidentally preventing your website from being indexed all together!

There are a lot of posts out there that talk about a Robots.txt file, how to set one up, etc. Unfortunately, many of these posts don’t apply specifically to WordPress and can often get confusing when you are trying to implement one for your website. Others tend to leave out a lot of important information that most people should have prior to attempting to add a Robots.txt file to their blog’s root directory.

Ask Apache is offering a Robots.txt file and updated Robots.txt file that you can download and use for your blog. You’ll of course want to customize it to meet your needs, including adding any directories that you have created since installing the base WordPress install that you don’t want them to be indexed.

I recommend you bookmark these posts if you plan to tackle adding a Robots.txt to your blog at some point in the future. Both posts provide a lot of valuable information and, as mentioned above, includes actual templates you can use for your website.

  • Leave a Comment
  • Search Engine Optimized (SEO) WordPress Theme Gallery

    When picking out a new WordPress theme, there are a lot of factors to consider. Do you want a 2-column theme or a 3-column theme? What about AJAX comments? Maybe you want specific colors? No matter what your preference is, one of the main factors you should probably consider is how search engine friendly your theme is going to be.

    There are a lot of WordPress SEO techniques that theme designers can use when creating their theme to help optimize it for search engines, and several authors have taken the time to do so. Here are what I feel are the best search engine optimized WordPress themes:

    2-Column Search Engine Optimized WordPress Themes

    Sodelicious Theme

    Sodelicious Theme

    Source | Download | Demo

    Modern Blue Green Theme

    Modern Blue Theme

    Source | Download | Demo

    YourBlog 2.0 Theme

    YourBlog 2.0 Theme

    Source | Download | Demo

    PassionDuo Theme (3 Colors)

    Green: Demo | Download, Red: Demo | Download, Blue: Demo | Download

    Yakuter Tema 2.0 Theme

    Yakuter Tema 2.0 Theme

    Source | Download | Demo

    3-Column Search Engine Optimized WordPress Themes

    Big Blue Theme

    Big Blue Theme

    Source | Download | Demo

    Redie 3.0 Theme

    Redie 3.0 Theme

    Source | Download | Demo

    Ambient Glo Theme

    Ambient Glo Theme

    Source | Download | Demo

    ProSense Theme

    ProSense Theme

    Source | Download (Orange, Blue, and Gray) | Demo

    Cutline Theme

    Cutline Theme

    Source | Download | Demo

    Have a search engine optimized WordPress theme you’d like to see included in this list? Feel free to comment below with a link and I’ll see about it getting added!

  • Leave a Comment
  • Prevent Duplicate Pages with Permalink Redirect

    There are only a select few WordPress plugins that I would consider “must-haves” for all types of blogs, no matter what the blog is about. Whenever I set up a new WordPress blog for someone, the first plugin I install is called Permalink Redirect. Everything can be controlled from within the Options panel, and it works out of the box without anything manual to set up.

    What does this plugin do? If you are using the most current version of this plugin, you’ll see that it is able to manage several different aspects of your WordPress blog. Its main function is to do a 301 redirect for all of your pages.  This means that it will redirect any alternate URL’s to the URL of your choosing. For example, this plugin automatically does the following for this blog:

    Wrong URL(s):

    http://www.wphacks.com

    http://www.wphacks.com/

    http://wphacks.com

    Correct URL:

    http://wphacks.com/

    If a reader enters one of the above incorrect URL addresses trying to reach my homepage, it will automatically redirect them to the correct homepage for this blog.  Your browser already does this you say?   That is probably correct. Most 2nd generation web browsers do in fact do this for you, but search engine spiders don’t use web browsers. Instead, search engine spiders would actually count these as 4 different web pages and could penalize you for having duplicate content.

    This plugin will also do the same 301 redirects for all individual webpages, as well as give you the option of manually redirecting certain pages if you change the post slug of that particular post or page for whatever reason.

    One often forgotten feature is the ability to redirect your posts if you were to change the permalink structure of your blog. For example, if you were to start out with the popular URL structure that many WordPress users use for their blog that includes the dates of the posts in the URL, you could later switch to a different structure (such as removing the date information to improve your search engine rankings) and redirect it with this plugin. It will automatically redirect all of your old posts to your new URL structure and make a flawless redirection for any old inbound links, as well as guide the search engine spiders to index your new post structure. An example of how this would look if I did this for this blog:

    Old URL:

    http://wphacks.com/2007/10/06/wordpress-plugins/

    New URL:

    http://wphacks.com/wordpress-plugins/

    What about feed redirection?  Many bloggers now take advantage of the amazing features Feedburner has to offer by setting up their blogs feed with them.   This can prove to be a bit of a headache to adjust all of your blog’s feeds to point to your new Feedburner feed, so this plugin will also take care of this for you. When in the options panel, simply enter your Feedburner feed address and save, and any time someone clicks on your default feed, they will automatically be redirected to your new Feedburner feed. This will help ensure you have an accurate count of feed subscribers, rather than splitting them up on multiple feeds.

    Overall, I am a huge fan of this WordPress plugin, and encourage anyone with a WordPress blog to use it. Now that tags are built into the default install of WordPress, I’m hoping this will be the next feature to be built into the software. It is definitely crucial in my opinion.

  • Leave a Comment