Rails, Exceptions, Hoptoad, and Annoying User Agents

January 06, 2009

Hoptoad is an excellent service that allows you to funnel all of your Rails exceptions to a central location so you can splice, dice, and fix them. I’ve noticed however that some web spiders mangle valid URLs into invalid ones and rather than deal with all the possible permutations it would be a lot nicer to simply have Hoptoad ignore them completely.

In my latest Rails app I noticed several web spiders that were mangling valid URLs into invalid ones. The combinations seemed to be completely random. Not every spider does this so I assume some are just poorly written or try permutations in hopes of finding other pages or are just confused.

Hoptoad doesn’t (currently) let you ignore exceptions raised by certain user agents. I wrote a patch that lets you do this and it’s being worked into the hoptoad plugin itself. It allows you to do this in config/initializers/hoptoad.yml:

HoptoadNotifier.configure do |config|
  config.api_key = 'your-api-key-here'
  config.ignore_user_agent << /Yahoo! Slurp/
  config.ignore_user_agent << /libwww-perl/

You can also pass in a string for an exact match if that’s easier for you. User agents are processed in the order you add them so take some care to optimize this list if necessary.

You can download the patch here and read the discussion/ticket here.