Hoptoad is an excellent service that allows you to funnel all of your Rails exceptions to a central location so you can splice, dice, and fix them. I’ve noticed however that some web spiders mangle valid URLs into invalid ones and rather than deal with all the possible permutations it would be a lot nicer to simply have Hoptoad ignore them completely.
In my latest Rails app I noticed several web spiders that were mangling valid URLs into invalid ones. The combinations seemed to be completely random. Not every spider does this so I assume some are just poorly written or try permutations in hopes of finding other pages or are just confused.
Hoptoad doesn’t (currently) let you ignore exceptions raised by certain user agents. I wrote a patch that lets you do this and it’s being worked into the hoptoad plugin itself. It allows you to do this in
HoptoadNotifier.configure do |config| config.api_key = 'your-api-key-here' config.ignore_user_agent << /Yahoo! Slurp/ config.ignore_user_agent << /libwww-perl/ end
You can also pass in a string for an exact match if that’s easier for you. User agents are processed in the order you add them so take some care to optimize this list if necessary.