I have previously released EPiRobots which is a generic robots.txt handler for your EPiServer CMS 6 R2 site. However I was working on an EPiServer Commerce project when I noticed a bug in EPiRobots so have released v1.0.1 onto the EPiServer Nuget Feed. You can see all the details below.
When clicking the "Manage robots.txt content" in admin mode you get a 404 error.
EPiRobots uses embedded resources to serve a file called "AdminManageRobotsTxt.aspx" in order to edit the robots.txt served for site. However EPiServer Commerce routes all requests for *.aspx at the site root to EPiServer Commerce in order to display product information. This meant the requests to virtual path used for accessing the plug in used to edit robots.txt were never routed and a 404 was being thrown.
I have now moved vurtual path of the embedded resource location to "/util/AdminManageRobotsTxt.aspx". This ensures that a 404 is not thrown and is a safe location as EPiServer uses this.
I'd be happy to hear any feedback on the comments below or @davidknipe.