Palegoldenrod

The Problem

Scenario: You have an ASP.NET 2.0 website in IIS 6.0 that receives both http and http requests. Some people might link to the https versions of the page, but you really want Google (and other search engines) to only crawl http versions. There are relative links in the site, so you know that as soon as Google crawls one https link it is going to pick up the whole site. So as well as https links in the Google search results, you may also now have to contend with lower rankings because Google thinks there is duplicate content on the site (it sees the page in both http and https format).

You can see that Google recommends that you have a different robots.txt file for http and https, so that Google knows not to crawl the https version of your site. But how can you get IIS to serve up different versions of robots.txt?

The rest of this article is available at KleeneCode.net, where Amos and I are now posting all of our tech posts. You can read the remainder of this article here