So maybe you are wondering, why should I ignore some routes? And the simple answer is: Because most of the time, different search engines/crawlers/bots will try to index your site and they will request for specific files. For example, Google will try to access the following file: http://yoursite.com/robots.txt. The robots.txt file is used to to give instructions about your site to web robots. More info can be found here. Up to now, everything sounds great, but if your site does not have robots.txt file, the ASP.NET MVC framework will raise an exception upon a request like the above. So your log, where the unhandled exceptions are reported can be flooded with messages like:
System.Web.HttpException : A public action method 'robots.txt' could not be found
on controller 'YourPage.Controllers.YourController'.
Obviously, there are two ways to handle this – provide a dummy robots.txt file or instruct the ASP.NET MVC framework to ignore such URL’s. I prefer the second approach. In this post I will try to summarize all the URL’s that should be ignored by the ASP.NET application. So, here they are:
Here is the code, taken from the Global.asax.cs file that should ignore some of the URL’s outlined above:
I will update the list above as soon as I find a new URL that should be ignored, so stay tuned. As well you are welcome to contribute to this list by posting a comment.