Stephan Zahariev's Blog

Routing is a key feature from the ASP.NET MVC. If you are not familiar with this concept, take a look here, because I’m not going to explain it. I’m going to share some routes that you should ignore in your configuration.
So maybe you are wondering, why should I ignore some routes? And the simple answer is: Because most of the time, different search engines/crawlers/bots will try to index your site and they will request for specific files. For example, Google will try to access the following file: http://yoursite.com/robots.txt. The robots.txt file is used to to give instructions about your site to web robots. More info can be found here. Up to now, everything sounds great, but if your site does not have robots.txt file, the ASP.NET MVC framework will raise an exception upon a request like the above. So your log, where the unhandled exceptions are reported can be flooded with messages like:
System.Web.HttpException : A public action method 'robots.txt' could not be found
on controller 'YourPage.Controllers.YourController'.
Obviously, there are two ways to handle this – provide a dummy robots.txt file or instruct the ASP.NET MVC framework to ignore such URL’s. I prefer the second approach. In this post I will try to summarize all the URL’s that should be ignored by the ASP.NET application. So, here they are:

• ~/robots.txt
  
• ~/sitemap
  
• ~/sitemap.gz
  
• ~/sitemap.xml
  
• ~/sitemap.xml.gz
  
• ~/google_sitemap.xml
  
• ~/google_sitemap.xml.gz
  
• ~/favicon.ico
  
• ~/mobile
  
• ~/iphone
  
• ~/apple-touch-icon.png
• ~/sitemap.txt
• ~/sitemap.rss

You need to ignore the routes above, ONLY if your site do not uses such a file. If you are not familiar with the sitemaps concept,  here is a quick overview that answers to most of the questions.
Here is the code, taken from the Global.asax.cs file that should ignore some of the URL’s outlined above:
I will update the list above as soon as I find a new URL that should be ignored, so stay tuned. As well you are welcome to contribute to this list by posting a comment.