Receiving Incoming Email in Rails 3 – choosing the right approach

When it comes to sending mail in Rails there are plenty of solutions and the best practices are fairly widely known. There are also a number of third party systems that can help you out. Receiving email in Rails however is a slightly less documented prospect.

Here I will cover some of the basic options for receiving email with some of the advantages and disadvantages. In later articles I will cover how to set up some of the solutions including the server. In each case I will also give a small example showing how you can find a user from the message’s to field update another from the message body. I don’t want to get too into the setup specifics of each approach at this point, instead I want to point out the alternatives and how you can make use of each. From what I can tell there are four main alternatives:

  • The ‘Official Way’ – using a mail server and script/rails runner
  • Using a mail server and cURL
  • Polling using IMAP/POP3
  • Use a Service Provider

It should be noted that I am the creator of one of the service providers (CloudMailin) however I appreciate that not all people want to use external services or have different needs and I am trying to make this article as objective as possible. Having said that if you do have comments please feel free to contact me.

Receiving Email the ‘Official Way’

The rails documentation is pretty sparse on incoming emails however the guides do contain a small snippet. Firstly you need to create a receive method in your ActionMailer. Something like the following for our example:

class MyMailer < ActionMailer::Base
  def receive(message)
    for recipient in
      User.find_by_email(recipient).update_attribute(:bio, message.body)

As you can see the ActionMailer class is quite simple, then all that is left is to wire up your server so that any incoming email is sent directly to ActionMailer. This can be done by making sure that your mail server executes the following command:

app_dir/script/rails runner 'MyMailer.receive('.

This approach has some serious disadvantages though, especially when it comes to scalability. Firstly every time you receive an email you are spawning an new instance of your environment with script/rails. This is a nightmare in itself. Along with this you also need a copy of your app on the same server as the mail server. So you either have to add the mail server to your app server or you need another server and copy of your app running for the mail. You also have the hassle of setting up a dedicated mail server just for the purpose of receiving these incoming emails.

The same approach using cURL

In order to improve this method it is possible to remove the call to script/rails runner and replace it with a call to the web app via HTTP using cURL. Using this method when a new email arrives the following is called:

ruby receiver.rb

Then we create our receiver something like the following:

# note the backticks here execute the command
`curl -d "message=#{}" http://localhost/incoming_messages`

Update: In the comments it turns out that some people have reported problems with this method. You may need to escape the content so that your app receives the message correctly. The following method should help:

require 'cgi'
# note the backticks here execute the command
`curl -d "message=#{CGI.escape(}" http://localhost/incoming_messages`

You could of course remove Ruby from the mix here entirely but using a Ruby script allows you to perform any processing if you want to in a more complex example. cUrl -d will send the request as application/x-www-form-urlencoded but you could also send the data multipart/form-data if you wish.

You can then simply create a normal controller and use the create method to receive your email as an HTTP POST. Something like the following:

def create
  message =[:message])
  for recipient in
      User.find_by_email(recipient).update_attribute(:bio, message.body)

This method has the advantage of being a little more scalable as nothing really changes in terms of your app. You simply receive the message over HTTP like any other form post or file upload. You may want to opt to move the processing out to a background job though if you are doing anything complex with the message. You will still however need to install and setup your own mail server.

Using a Third Party

In the last example we received email via an HTTP Post as a webhook. There are a couple of options for taking the setup and monitoring stress out of receiving mail in this manor without having to install an configure a mail server. Two of the options here are CloudMailin and smtp2web.

CloudMailin is currently in free beta and allows you to register to receive email via HTTP Post. The system was designed to be scalable and provide some additional features like delivery logs to make sure your app is receiving the emails. That’s enough about that one as I don’t want to be biased.

smtp2web is a google app engine instance that can be used to achieve a similar goal. It make use of app engines ability to receive email and then forwards the messages on to your web app.

Both of these options are designed to operate in ‘the cloud’ and don’t require you to own or setup a mail server to do the work. You will again probably want to make sure that you move processing over to a background worker if you have anything complex to do so that the processing doesn’t take up resource that should be serving your app to your customers.

Polling Using IMAP or SMTP

Finally this solution makes sense when you need to collect messages from an existing mailbox. You don’t have to own your own mail server but you will need to be able to run cron or a daemon to collect mail at regular intervals.

Although you could roll your own collector there are a couple already out there. Take a look at mailman for example. This approach can either rely on direct acces to your blog or can again POST via HTTP.

I will also look to write a separate post on MailMan as I think the power offered by MailMan is a worth a blog post in itself. Although there will be a delay with any polling as you can only poll every few minutes, in some situations using an existing mailbox is the only option.

Although this was brief, it should have given a quick introduction into some of the approaches available (I’m sure there are more too). I also plan to write a number of follow up articles showing how to implement options described here. If you have any advice, an alternative option or even an approach you would prefer to see covered first then please jump in and comment. Again if you have any comments on CloudMailin please let me know on here, twitter or via email at blog-comments [you know what goes here]

Automatically prepending url’s with http://

Recently we added functionality that allowed users to include links to images that they uploaded to one of our sites. In order to make the experience as easy as possible for users we allowed them to enter the url with or without the protocol (http:// or https://).

In order to make sure that any of our models that stored the information would always return a link with the protocol in it I wanted to create a simple mixin that would override the existing link method returned from the database and prepend http:// to it if it needed to.

Checking for the protocol and inserting it
This is actually quite a simple method. The following code was used to override the source_url method that was returning the link from the database.

def source_url
  link = super
  "#{link.match(/(http|https):\/\//i) ? '' : 'http://'}#{link}"

Since I was going to add this to a number of models it made sense to convert this to a mixin that could be used on any of the modules.

module Protocolize
  def self.included(klass)
    klass.class_eval do
      def self.protocolize(link_method)
        define_method link_method.to_sym do
          link = super()
          return nil if link.blank?
          "#{link.match(/(http|https):\/\//i) ? '' : 'http://'}#{link}"

This can then be called using the following in your model:

include Protocolize
protocolize :method_name

Notice that you have to explicitly call super() with params and not just super when you use it within define_method. If you don’t you will get the following error:

implicit argument passing of super from method defined by define_method() is not supported. Specify all arguments explicitly.

Just a tiny snippet that might be useful to people to ensure their links work correctly.

Rails 3, Rake and url_for

Before I start I just want to make it clear that I know the arguments against using url_for in models and even in rake tasks. Sometimes however it makes sense to use_url for in a rake task. In my case I am trying to query another site’s api which requires the URI of the page on my site that I want to gather information about.

The approach in Rails 2.x

task :collect_stats => :environment do
  include ActionController::UrlWriter

  default_url_options[:host] = ''
  url = url_for(:controller => 'foo', :action => 'bar')

Notice that because there is no current request you have to specify the


as the helper has no idea what the host will be otherwise.

Doing the same thing in Rails 3

The following code does the same thing in Rails 3.

task :collect_stats => :environment do
  include ActionDispatch::Routing::UrlFor
  #include ActionController::UrlFor  #requires a request object
  include ActionController::PolymorphicRoutes
  include Rails.application.routes.url_helpers

  default_url_options[:host] = ''
  url = url_for(post)

There are two key points to notice here.

  1. The first is that I have included ActionDispatch::Routing::UrlFor rather than ActionController::UrlFor. The latter requires a request object and will attempt to automatically fill in the host name. Since we are in a rake task there is no request and the method will fail.
  2. The second thing is that I have also included two additional includes. The will allow you to work with polymorphic routes and named routes, giving a bit more flexibility.

Just a short snippet that might be of use to people but if there are any improvements out there then please let me know and I will update this. You can of course hard code the routes but there are scenarios where it makes much more sense to make use of the helpers provided, especially when using polymorphic routes.

Update: 08/06/2010

In the comments Jakub has stated that in the latest version of Rails you don’t need to include the polymorphic routes.

include ActionController::PolymorphicRoutes

Render ‘Rails Style’ Partials in Sinatra

We love Sinatra. Not only does it make a great framework in its own right but in addition it can be used to mimic parts of rails in a real simple environment for front-end designers. Instead of having to get them set up and explain the whole of rails they just get a nice simple app to work on without having to worry about creating different controllers or even models.

Although there is not a 1 to 1 translation between a rails app and a sinatra one, it does allow these developers to work with things like haml in a really easy to work with environment.

One of the features that I was asked for recently though was “How do you render a partial in sinatra?”

Rendering Partials in Sinatra

Sinatra is a super-lightweight framework. Because of this it doesn’t have the notion of partials built into it. However, a partial, in its simplest form, is nothing more than a call out to render the template as a string and then embed that string into your page.

A quick look at the sinatra sites FAQs shows that partials can be rendered in the following way in erb.

<%= erb(:mypartial, :layout => false) %>

In haml you could use exactly the same thing but call haml like so.

= haml(:mypartial, :layout => false)

Notice that

:layout => false

is set to ensure that the layout is not also rendered.

Going a little further

The FAQs also recommend using the code in the following gist.

The code shows a helper method called partial. This helper method can be used to render a partial from your code. The helper also allows you to pass collections and is a really cool and useful piece of code.

Making things work the rails way

The above helpers are great and really useful for sinatra. However, what if you want to render a partial the ‘rails way’? In our situation we were using sinatra as a mock up of what would eventually be brought into a rails app. Rails allows partials to be included like so:

<%= render :partial => 'partial_name' %>

By overriding the built in render method in Sinatra it is actually possible to mimic the rails partials. I came up with the following helper to quickly mock things up. The helper checks to see if the first argument passed to is a hash and if that contains they key :partial. If so it renders the partial, if not it just uses the default render method.

  helpers do
    def render(*args)
      if args.first.is_a?(Hash) && args.first.keys.include?(:partial)
        return haml "_#{args.first[:partial]}".to_sym, :layout => false

The helper could easily by extended to allow for collections etc but for now it does the job. Any better solutions?

Using Rack::Rewrite to remove the www

Rack::Rewrite is a pretty useful tool. It allows you to change the location of a url or perform 301 or 302 redirects using rack. The real benefit of this is that the logic of your routes is contained within your app and server agnostic. Sure there is a performance hit compared to using mod-rewrite etc. in Apache, Nginx and so on but we feel its worth it in most cases. If you are using caching systems like varnish or squid the redirect can be cached too.

Using Rack::Rewrite to go NoWWW

There has been a bit of a movement for a while on the web to deprecate www. in front of domain names called no-www. As the site says the title www isn’t really of any use anymore. As well as the no-www movement having a single host can also be useful for things like analytics and cleaning up search results.  There are a couple of rails solutions to doing this including a peice of rack middleware at this site

If you are already using Rack::Rewrite though you may as well just use that to make the change. The following is a rule to make the no-www change using rack middleware.

r301 /.*/, {|path, rack_env| "http://#{rack_env['SERVER_NAME'].gsub(/www\./i, '') }#{path}" },
    :if => {|rack_env| rack_env['SERVER_NAME'] =~ /www\./i}

ID free pretty permalink based URL’s in Rails

Slug based URLS

I love the way that rails allows you to make use of REST and create meaningful urls for your web applications. The only problem is that, although these urls are meaningful, having to use a model’s id in the URL can be pretty ugly.

Replacing the model’s id with a meaningful title can be useful to both users and for SEO purposes. A lot of approaches in rails make use of the fact that calling to_i on a string will start at the begining of a string and work forward until there are no numbers like so:

"1".to_i # => 1
"123four".to_i => 123

This has led to a number of rails plugins that allow you to make id and permalink based urls like the following:

acts_as_friendly_param by Chris Farms is one of the plugins that you can use to achieve this.

Rails Slugs Are Bad

Slugs are bad kids. Okay? (There’s actually nothing wrong with slugs that contain an ID and while I don’t have any objection to creating URL’s like the above for some things they just seem unnecessary).

There is however another approach that allows you to use a unique reference for each record in your models. If you store a unique permalink for each model then you can use that link in your models finder in order to find that record like so:


The only problem with this approach is you have to edit all of your code to use the different finder method. Enter the slugs_are_bad plugin. I created the plugin to override rails’ default finder methods. The plugin is based on a couple of other plugins acts_as_friendly_param by Chris Farms and permalink_fu by Technoweenie but was adapted to suit my needs a little more.

Using the plugin

The slugs_are_bad plugin forms a drop-in replacement to create pretty, id-free, urls. In most cases you don’t have to make any changes from the default rails scaffold apart from adding the slugs_are_bad plugin.

  slugs_are_bad(:permalink_attribute, :generate_from)

  # ie. slugs_are_bad(:permalink, :title)
  # will automatically generate a slug-less permalink from the title attribute and store it in the model.

Then in your view
  link_to 'User', => 'foo bar') # nothing new needed here
  # Generates /users/foo-bar instead of /users/1

  # create the user as usual
  User.create!(:name => 'foo bar')

  # To find the model with its nothing else is required
  User.find(params[:id]) # nothing needed here (where id will be 'foo-bar')

  # you could also manually specify the permalink in this instance if wanted.
  User.create!(:name => 'foo-bar', :permalink => 'foo')

However, there are still a few things missing. The plugin is not quite as flexible as I would like it to be and also there are no tests for it at the moment. I have a set of tests that I use in a production site to ensure the plugin works for that project but I havent created any tests for the standalone plugin itself yet. I should also note that the plugin was been created for rails 2.3.x, I know that rails 2.3 is obsolete now :P but hey some people might find it useful.

If anyone wants to expand upon the plugin then fork away on github and ping me the changes.

Using the new Rails 3.0 Gem bundler with Passenger, Mongrel and Heroku

UPDATE 19th Feb 2010: Bundler is moving pretty fast! For the most up to date information I’d checkout and specifically this gist ( to find out the latest!

Recently we were unsure whether we would be deploying a site to our own hosted system or heroku. I love heroku but there are times when it just doesnt suit the project and a bit more fine grained control is nessesary.

In order to use heroku it was suggested that you move over to the new gem bundler that Yahuda has been working on as part of rails 3.0. However it seems there are a couple of different ways to get the bundler running. The default recomended way works great for mongrel and heroku but didn’t play so nice with passenger.

The default is to place all of the following code into config/preinitializer.rb:

require "#{File.dirname(__FILE__)}/../vendor/bundler_gems/environment"

class Rails::Boot
  def run

  def extend_environment
    Rails::Initializer.class_eval do
      old_load = instance_method(:load_environment)
      define_method(:load_environment) do
        Bundler.require_env RAILS_ENV

After a bit of searching around it seems that in some spawn methods this does not work great with passenger.

The solution

Instead use the following:

In your config/preinitializer.rb just include this part

require "#{File.dirname(__FILE__)}/../vendor/bundler_gems/environment"

Then in config/boot.rb place this just before the last Rails.boot! line like so:

# for bundler
class Rails::Boot
  def run

  def extend_environment
    Rails::Initializer.class_eval do
      old_load = instance_method(:load_environment)
      define_method(:load_environment) do
        Bundler.require_env RAILS_ENV

# All that for this:

That should allow you to easily boot from either server and works great with heroku.

Thanks to Mathew Todd for giving the solution based upon gem cutters commit here.

Random, Previous and Next entries from Active Record models using offset

I have seen a couple of entries floating around on the internet relating to how to obtain a random active record entry or the next or previous active record entries. Many of the examples are actually quite inefficient and although they’ll get the job done they’re probably not best for the poor server. I’m certainly not an expert so any corrections on my methods are more than welcome. Offset is often underused feature by rails developers, in my experience, so I thought I’d post a couple of useful use cases for offset.

Obtaining a random record

One of the ways that seems to appear sometimes is to make use of the MySQL database and some custom ordering to return a list of records in a random order and then just limit the result to the first. This would be done using something like the following.

Entry.find(:all, :limit => 1, :order => "RAND()")

This is not efficient in MySQL. Done this would cause your table to take every record and then generate a random number for it. The time that this takes would therefore be proportional to the number of rows in your table! You also wouldn’t really want to say Entry.find(:all).rand because the entire record set would be returned before a random entry is chosen.

Although it might seem like it would take a little longer as there are two calls my preferred solution would be to add the following method to you model (entry.rb)

  def self.random(options = {})
    count = self.count()
    self.find(:first, options.merge(:offset => rand(count)))

This code will find the count of all of the rows in the table and then use this to choose a record with a random offset. The offset is used to make sure this works when records are deleted. While this could also probably be done in the database with the offset being a select statement any named scopes would not be present on the embedded select and you might be choosing a random number from a larger set than is actually available to the query introducing a bias in the random number chosen. Clearly making two calls carries a tiny amount of risk if the number of items changes though but in the worst case you will simply most likely up returning the most recent entry and this method works across any database as there is nothing specific in here.

Getting the next and previous entries

The use of offset can also be applied to get the next and previous records from the database too. I find using a named scope to act as a class method on the model works great.

named_scope :previous, lambda { |i| {:conditions => ["#{self.table_name}.id  "#{self.table_name}.id DESC"} }
      named_scope :next, lambda { |i| {:conditions => ["#{self.table_name}.id > ?",], :order => "#{self.table_name}.id ASC"} }

This specifies the table name of the model automatically to avoid any conflicts that might occur. I even wrapped this one into a tiny little plugin that can be used to save typing this out for different models. The plugin can be found on github


This should have shown a few pretty cool uses for offset but I’d also be interested in hearing responses to the approach for obtaining random records. Does anyone do things differently and how is that method better?


if you need a library to help you pull out more than one random entry then it is worth taking a look at The plugin is pretty lightweight but should help make things a little easier.

Ruby Gems and OSX 10.6 (Snow Leopard)

I recently upgraded to Snow Leopard. I had to fight the urge to install sooner because we had some big projects underway at work and couldn’t afford any downtime.

Due to not having time to do a complete re-install I simply did the upgrade and I must admit I am pretty impressed with how smoothly it went. When I get some more time I’ll do a clean install but I’m happy with things for now.

One of the things I am finding however is that a lot of gems that required native extensions need to be rebuilt. For a number of these things are quite obvious and its just a case of either updating or removing the gem and re-installing.

Autotest however wasn’t quite so obvious. Some gems install dependancies that required rebuilding and its not always 100% obvious like the following:

/Library/Ruby/Gems/1.8/gems/sys-uname-0.8.3/lib/sys/uname.bundle: dlopen(/Library/Ruby/Gems/1.8/gems/sys-uname-0.8.3/lib/sys/uname.bundle, 9): no suitable image found.  Did find: (LoadError)
/Library/Ruby/Gems/1.8/gems/sys-uname-0.8.3/lib/sys/uname.bundle: no matching architecture in universal wrapper - /Library/Ruby/Gems/1.8/gems/sys-uname-0.8.3/lib/sys/uname.bundle
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:31:in `require'
from /Library/Ruby/Gems/1.8/gems/autotest-fsevent-0.1.1/lib/autotest/fsevent.rb:3
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:36:in `gem_original_require'
from /Library/Ruby/Site/1.8/rubygems/custom_require.rb:36:in `require'
from /Users/stevesmith/.autotest:2
from /Library/Ruby/Gems/1.8/gems/ZenTest-4.1.3/lib/autotest.rb:195:in `load'
from /Library/Ruby/Gems/1.8/gems/ZenTest-4.1.3/lib/autotest.rb:195:in `initialize'
from /Library/Ruby/Gems/1.8/gems/ZenTest-4.1.3/lib/autotest.rb:194:in `each'
from /Library/Ruby/Gems/1.8/gems/ZenTest-4.1.3/lib/autotest.rb:194:in `initialize'
from /Library/Ruby/Gems/1.8/gems/autotest-rails-4.1.0/lib/autotest/rails.rb:7:in `initialize'
from /Library/Ruby/Gems/1.8/gems/ZenTest-4.1.3/lib/autotest.rb:138:in `new'
from /Library/Ruby/Gems/1.8/gems/ZenTest-4.1.3/lib/autotest.rb:138:in `run'
from /Library/Ruby/Gems/1.8/gems/ZenTest-4.1.3/bin/autotest:55
from /usr/bin/autotest:19:in `load'
from /usr/bin/autotest:19

The gem sys-uname makes use of native extensions but is obviously not reinstalled as autotest is its just assumed to already exist. In this case simply doing

sudo gem uninstall sys-uname
sudo gem install sys-uname

will bring it back. I guess this isn’t the most tricky thing to figure out but I thought I’d post it in case it was of use to anyone.

Fragment Cache Expiration in Ruby on Rails

A few days ago I was trying to implement fragment caching with a timed expiration using memcached in rails. I needed a system to cache the results of querying a large number of rss feeds that were being aggregated on the page. Whilst I could have created a system to do this offline and store the results it was easier to just fragment cache and show a stale cache whilst new results were found.

When manually caching or in action caching you simply pass :expires_in => 5.minutes to tell memcached to automatically expire the page after 5 minutes. However this didn’t seem to work with fragment caching, after a tiny bit of digging and error I realised that in order to specify the options hash you have to first specify a key. Since the key is generally made up of the request url I simply used this and appended a unique id for the fragment. Obvious when you think about it I guess!

cache("#{request.url}?fragment", {:expires_in => 60.seconds}) do