I have previously released EPiRobots which is a generic robots.txt handler for your EPiServer CMS 6 R2 site. However I was working on an EPiServer Commerce project when I noticed a bug in EPiRobots so have released v1.0.1 onto the EPiServer Nuget Feed. You can see all the details below.
EPiRobots is an EPiServer plug in that handles delivery and modification of the robots.txt file for your EPiServer CMS 6 R2 site(s). It has no dependency on any page types, requires no .config modifications and should work in any EPiServer CMS 6 R2 deployment scenario.
Recently I spoke to a client who had recently migrated to EPiServer from another CMS. They'd noticed that the number of internal broken links was going up rather than down after the migration. I thought that this was strange as EPiServer maintains all internal links by reference so if a page is moved in the page tree then it's link is automatically updated.