Receiving Incoming Email in Rails 3 – choosing the right approach

When it comes to sending mail in Rails there are plenty of solutions and the best practices are fairly widely known. There are also a number of third party systems that can help you out. Receiving email in Rails however is a slightly less documented prospect.

Here I will cover some of the basic options for receiving email with some of the advantages and disadvantages. In later articles I will cover how to set up some of the solutions including the server. In each case I will also give a small example showing how you can find a user from the message’s to field update another from the message body. I don’t want to get too into the setup specifics of each approach at this point, instead I want to point out the alternatives and how you can make use of each. From what I can tell there are four main alternatives:

  • The ‘Official Way’ – using a mail server and script/rails runner
  • Using a mail server and cURL
  • Polling using IMAP/POP3
  • Use a Service Provider

It should be noted that I am the creator of one of the service providers (CloudMailin) however I appreciate that not all people want to use external services or have different needs and I am trying to make this article as objective as possible. Having said that if you do have comments please feel free to contact me.

Receiving Email the ‘Official Way’

The rails documentation is pretty sparse on incoming emails however the guides do contain a small snippet. Firstly you need to create a receive method in your ActionMailer. Something like the following for our example:

class MyMailer < ActionMailer::Base
  def receive(message)
    for recipient in message.to
      User.find_by_email(recipient).update_attribute(:bio, message.body)
    end
  end
end

As you can see the ActionMailer class is quite simple, then all that is left is to wire up your server so that any incoming email is sent directly to ActionMailer. This can be done by making sure that your mail server executes the following command:

app_dir/script/rails runner 'MyMailer.receive(STDIN.read)'.

This approach has some serious disadvantages though, especially when it comes to scalability. Firstly every time you receive an email you are spawning an new instance of your environment with script/rails. This is a nightmare in itself. Along with this you also need a copy of your app on the same server as the mail server. So you either have to add the mail server to your app server or you need another server and copy of your app running for the mail. You also have the hassle of setting up a dedicated mail server just for the purpose of receiving these incoming emails.

The same approach using cURL

In order to improve this method it is possible to remove the call to script/rails runner and replace it with a call to the web app via HTTP using cURL. Using this method when a new email arrives the following is called:

ruby receiver.rb

Then we create our receiver something like the following:

# note the backticks here execute the command
`curl -d "message=#{STDIN.read}" http://localhost/incoming_messages`

Update: In the comments it turns out that some people have reported problems with this method. You may need to escape the content so that your app receives the message correctly. The following method should help:

require 'cgi'
# note the backticks here execute the command
`curl -d "message=#{CGI.escape(STDIN.read)}" http://localhost/incoming_messages`

You could of course remove Ruby from the mix here entirely but using a Ruby script allows you to perform any processing if you want to in a more complex example. cUrl -d will send the request as application/x-www-form-urlencoded but you could also send the data multipart/form-data if you wish.

You can then simply create a normal controller and use the create method to receive your email as an HTTP POST. Something like the following:

def create
  message = Mail.new(params[:message])
  for recipient in message.to
      User.find_by_email(recipient).update_attribute(:bio, message.body)
    end
  end
end

This method has the advantage of being a little more scalable as nothing really changes in terms of your app. You simply receive the message over HTTP like any other form post or file upload. You may want to opt to move the processing out to a background job though if you are doing anything complex with the message. You will still however need to install and setup your own mail server.

Using a Third Party

In the last example we received email via an HTTP Post as a webhook. There are a couple of options for taking the setup and monitoring stress out of receiving mail in this manor without having to install an configure a mail server. Two of the options here are CloudMailin and smtp2web.

CloudMailin is currently in free beta and allows you to register to receive email via HTTP Post. The system was designed to be scalable and provide some additional features like delivery logs to make sure your app is receiving the emails. That’s enough about that one as I don’t want to be biased.

smtp2web is a google app engine instance that can be used to achieve a similar goal. It make use of app engines ability to receive email and then forwards the messages on to your web app.

Both of these options are designed to operate in ‘the cloud’ and don’t require you to own or setup a mail server to do the work. You will again probably want to make sure that you move processing over to a background worker if you have anything complex to do so that the processing doesn’t take up resource that should be serving your app to your customers.

Polling Using IMAP or SMTP

Finally this solution makes sense when you need to collect messages from an existing mailbox. You don’t have to own your own mail server but you will need to be able to run cron or a daemon to collect mail at regular intervals.

Although you could roll your own collector there are a couple already out there. Take a look at mailman for example. This approach can either rely on direct acces to your blog or can again POST via HTTP.

I will also look to write a separate post on MailMan as I think the power offered by MailMan is a worth a blog post in itself. Although there will be a delay with any polling as you can only poll every few minutes, in some situations using an existing mailbox is the only option.

Although this was brief, it should have given a quick introduction into some of the approaches available (I’m sure there are more too). I also plan to write a number of follow up articles showing how to implement options described here. If you have any advice, an alternative option or even an approach you would prefer to see covered first then please jump in and comment. Again if you have any comments on CloudMailin please let me know on here, twitter or via email at blog-comments [you know what goes here] cloudmailin.com

About these ads

About Steve Smith
Software developer (often ruby, rails but I enjoy loads of languages), semantic tech. fanboy, skydiver, all round geek. Owner of dynamic:edge (hire us) the makers of CloudMailin.com

23 Responses to Receiving Incoming Email in Rails 3 – choosing the right approach

  1. S. M. Sohan says:

    Your post has been linked at the Drink Rails blog as one of the useful Ruby on Rails blog posts of the day.

  2. Ed says:

    Great intro to receiving email with rails. Just the kind of tutorials i badly needed. As an aside if not for this article, i won’t have heard of ‘cloudmailin’. I certainly will sign up to beta test it. but i am non the less very interested in this seriesof tutorials. Can you start with recieving email via IMAP/POP3 using POST via HTTP. You can then follow up with mailman implemetation. I am interested in building a service that has the kind of email integration that posterous has, where people can post via email. So keep it coming.

    thanks.

    • Steve Smith says:

      I have a Blog post half written that covers making something like Posterous with Sinatra, MongoDB/GridFS and CloudMailin. Maybe I should get on with it :-)

  3. Mailman was just developed out of the 2010 Ruby Summer of Code for just that purpose. Check it out.

    http://github.com/titanous/mailman

    • Steve Smith says:

      I agree mailman is an awesome project and I will certainly follow up with more information about mailman as I have said in the post. I still think it’s important to know all of the options though and pick the best solution for your project, which might not always be Mailman.

  4. Jeff says:

    Thank you so much for posting this. Cloudmailin.com is exactly the type of service I’ve been looking for for several months. My host no longer allows procmail, which would be a nice, event-driven solution, so I ended up doing the less-scalable methods you mentioned above (along with a frequent cron task). I’d thought about starting a service like cloudmailin, but didn’t really have the resources to make it happen. Thanks, you’ve really filled a huge need for me.

  5. Maarten says:

    Thanks for pointing out the cURL approach.

    You can get rid of the receiver.rb by calling curl directly:

    curl -F “message=<-" http://localhost:3000/incoming_messages

    I now use Exim to deliver messages to my Rails app with a transport like this:

    my_rails_app:
    driver = pipe
    command = /usr/bin/curl -F 'message=<-' http://my_app_dom/incoming

    Easy!

  6. Pingback: A Step by Step Guide to Receiving Email in your Web Application with CloudMailin « Steve @ DynamicEdge

  7. Pingback: Receiving Test Driven Incoming Email for Rails 3 « Steve @ DynamicEdge

  8. Pingback: Receiving and Saving Incoming Email Images and Attachments with Paperclip and Rails 3 « Steve @ DynamicEdge

  9. Tobias Müller says:

    I tried the curl version, which is not working for me. What mail library are you using? The ‘mail’ gem?

    • Steve Smith says:

      Yes the ‘mail’ gem is https://rubygems.org/gems/mail where does your error occur?

      • Tobias Müller says:

        My problem is: I even get no error. The mail message object is simply not initialized. Here is the message param from the server console:
        Processing by IncomingEmailController#create as
        Parameters: {“message”=>”From tobmue@localhost.local Mon Jul 11 21:54:47 2011\nReturn-Path: \nX-Original-To: sell@localhost\nDelivered-To: sell@localhost.local\nReceived: from Tobias-Muellers-MacBook-Pro.local (localhost [127.0.0.1])\n\tby Tobias-Muellers-MacBook-Pro.local (Postfix) with ESMTP id 0F29FD9CEA3\n\tfor “, “Mon, 11 Jul 2011 21:54:47 0200 (CEST)\nMessage-ID: \nDate: Mon, 11 Jul 2011 21:54:46 0200\nFrom: tobmue localhost \nUser-Agent: Mozilla/5.0 (Macintosh”=>nil, “U”=>nil, “Intel Mac OS X 10.6″=>nil, “de”=>nil, “rv:1.9.2.18) Gecko/20110616 Thunderbird/3.1.11\nMIME-Version: 1.0\nTo: sell@localhost.local\nSubject: test2154\nContent-Type: text/plain”=>nil, “charset”=>”ISO-8859-15″}
        Is this correct? The message object is empty.

      • Steve Smith says:

        This isn’t something that I have seen but I have only used this method for very simple examples. I will have a go with some more complex examples and see if I can reproduce the error. It might be that url encoding is needed for example.

      • khojguru says:

        I am also getting params like :-

        {“Thu, 11 Aug 2011 16:24:29 0530 (IST)\nReceived: by vwl1 with SMTP id 1so1848759vwl.29\n for “=>nil, “boundary”=>”20cf3071c922f7c4cd04aa389d21\n\n–20cf3071c922f7c4cd04aa389d21\nContent-Type: text/plain”, “Thu, 11 Aug 2011 03:54:12 -0700 (PDT)\nMIME-Version: 1.0\nReceived: by 10.52.71.195 with SMTP id x3mr1432829vdu.453.1313060051993″=>nil, “charset”=>”ISO-8859-1\n\n– Amit Kumar Sharmaamit@khojguru.com\n\n–20cf3071c922f7c4cd04aa389d21–\n”, “Thu, 11 Aug 2011 03:54:11 -0700 (PDT)\nDate: Thu, 11 Aug 2011 16:24:11 0530\nMessage-ID: “13C LSR2Q@mail.gmail.com>\nSubject: HUH\nFrom: Amit Kumar Sharma \nTo: abc@winnoclean.com\nContent-Type: multipart/alternative”, “message”=>”Received: from mail-vw0-f42.google.com (mail-vw0-f42.google.com [209.85.212.42])\n\tby mail.winnoclean.com (Postfix) with ESMTPS id 1931F2FE5C\n\tfor “, “Thu,\n 11 Aug 2011 03:54:11 -0700 (PDT)\nReceived: by 10.52.111.232 with HTTP”=>nil}

      • khojguru says:

        Escaping does the trick. Thanks

      • Tobias Müller says:

        What do you mean with url encoding? Where should this happen?
        Thanks
        Tobias

      • Steve Smith says:

        For example I changed:

        `curl -d “message=#{STDIN.read}” http://localhost/incoming_messages`

        to:

        `curl -d “message=#{CGI.escape(STDIN.read)}” http://localhost/incoming_messages`

      • Tobias Müller says:

        Thank you Steve.
        I tried it in the controller which doesn’t work.
        Escaping in the receiver script works.

      • Tobias Müller says:

        You can encode directly with cURL. You can use ‘curl -F’ or ‘curl –data-urlencode’.
        If you want to receive attachments the curl approach has one disadvantage. The length of commandline parameters is restricted. If the mail is to big, curl will throw an error. What’s a good solution to receive attachments =< 5mb with a rails application that scales?

  10. Pingback: Receiving incoming emails in rails 3 | RoRDeveloper

  11. mreinsch says:

    I also just published an article on how we’re relaying email from Postfix to Rails using curl. It’s pretty similar to what you’re describing here, but I wanted to mention it because we’re also leveraging Postfix’s ability to defer message delivery in case of communication errors due to maintenance or other issues. See http://www.doorkeeperhq.com/developer/smtp-to-web-api for details.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 362 other followers

%d bloggers like this: