Receiving and Saving Incoming Email Images and Attachments with Paperclip and Rails 3

The last post showed how the use of Test Driven Development can help create Rails apps that can receive email offline that can be plugged into one of the incoming email solutions to receive email. The example was a movie poster model that took the subject of the email and added it as the title of that movie. The thing is what’s a movie poster without images? In this post I am going to cover how you can extract an image out of an email and attach it to a model using Paperclip.

We’ve already covered how to receive the email using CloudMailin and the app’s controller and pass it onto a mailer so I’m only going to cover the code we need in our mailer and models here.

Getting Paperclip setup

The first step is to get paperclip installed into our app. I would recommend following the instructions in the Paperclip readme here to get your app set up with Paperclip and test everything works with a normal controller and form. Our movie poster will need the following setup:

We start by adding the migration to add the Paperclip columns to the migration

t.string    :poster_file_name
t.string    :poster_content_type
t.integer   :poster_file_size
t.datetime  :poster_updated_at

We also need to add the magic Paperclip line to our Movie model

has_attached_file :poster, :styles => { :medium => "300x300>", :thumb => "100x100>" }

So far everything has been exactly the same as we would normally have to do if we were using the normal forms based approach to upload, next we’ll add the code to our mailer.

Setting up the Mailer to Receive Attachments

Normally when Rails receives an uploaded file it has to be through a multipart/form-data. When a Rails app receives these files it recognises the attachment parameter and behind the scenes a temporary file is created containing that content. This is to stop the attachment being stored in memory. This file is then extended by Paperclip to contain a few additional bits of information it’s original file name and the content type of the upload. Our email however is stored in memory and it’s attachments don’t get stored as files. For that reason we need to emulate the temporary file so that Paperclip can correctly work with out attachments.

In order to do this we can use the following mailer:

class MovieMailer < ActionMailer::Base
  # Create an attachment file with some paperclip aware features
  class AttachmentFile < Tempfile
    attr_accessor :original_filename, :content_type
  end

  # Called whenever a message is received on the movies controller
  def receive(message)
    # For now just take the first attachment and assume there is only one
    attachment = message.attachments.first

    # Create the movie itself
    Movie.create do |movie|
      movie.title = message.subject

      # Create an AttachmentFile subclass of a tempfile with paperclip aware features and add it
      poster_file = AttachmentFile.new('test.jpg')
      poster_file.write attachment.decoded
      poster_file.flush
      poster_file.original_filename = attachment.filename
      poster_file.content_type = attachment.mime_type
      movie.poster = poster_file
    end
  end
end

Here we can see the AttachmentFile class that fakes the parameters needed by Paperclip to work with the file. We then create the AttachmentFile class passing in a filename to be used for the temporary file. Then we write to the file, flush the contents and set the required parameters for Paperclip.

Now when we receive an email the first attachment is taken from the email and stored as our movie poster with by Paperclip.

If you don’t want to use a temporary file you can also use a StringIO object and see the same benefits.

In addition to this @pedrodelgallego pointed out another posible solution that extends a StingIO class anonymously using class eval using the following code:

attachment = message.attachments.first

file = StringIO.new(attachment.decoded)
file.class.class_eval { attr_accessor :original_filename, :content_type }
file.original_filename = attachment.filename
file.content_type = attachment.mime_type

Hopefully this will be of use for anyone trying to store attachments extracted from emails. The real benefit is that Paperclip doesn’t only work with images, you can store all sorts of attachments and push them to S3 if you need to. If anyone has any other suggestions or better ways to achieve this I’d also love to hear them.

Advertisements

Receiving Test Driven Incoming Email for Rails 3

One of the most frequent questions we are asked at CloudMailin is how can you develop your application and receives email offline, without opening ports, in a Rails app.

There are options that spring to mind. The first is to write a script that delivers a local email to your app simulating the response that you would get from an email server. The second solution is far simpler. You can perform test driven development and it can be really easy  get things up and running in development and even easier to move to production.

Here we are going to create a really simple example to take the subject of an email and create a model called movie poster from that subject. In a later post we will show how we can also use the attachments to create the images for the poster.

The First Step, a Controller to Receive the Email

In a previous article I highlighted a number of options for Receiving Email in Rails 3. Using some of the approaches you don’t even need to use a controller but in that case where email is delivered to your app via an HTTP POST the controller is needed. The controller doesn’t have to be complicated though. All we really want to do is pass a mail object to the receive method of our mailer. So supposing we have a mailer called MovieMailer we can do the following:

class IncomingMailsController < ApplicationController
   skip_before_filter :verify_authenticity_token
   
   def create
     # create a Mail object from the raw message
     movie_poster = MovieMailer.receiver(Mail.new(params[:message]))
     # if the movie poster was saved then send a status of 201 else give a 422 with the errors
     if !movie_poster.new_record?
        render :text => "Success", :status => 201, :content_type => Mime::TEXT.to_s
    else
      render :text => movie_poster.errors.full_messages.join(', '), :status => 422, :content_type => Mime::TEXT.to_s
   end
  end
end

The example above assumes that you are using at least Rails 3.0 but it can easily be converted to work with older versions of Rails and you could use the TMail gem rather than Mail if you need to.

Now we have this controller we can create a functional test to check this in the same way we would any other controller in our app (note I’m doing this the wrong way round for illustration you should really write your test first!). If we mock the response from the Mailer for now we can use something like this with Shoulda but you could use RSpec or any other testing framework just as easily:

class IncomingMailsControllerTest < ActionController::TestCase
   context "An IncomingMailsController" do
     context "on POST to :create with an invalid item" do
        setup do
           MoviePoster.any_instance.expects(:valid?).returns(false)
           post :create, :message => "From: #{@user.email}\r\nSubject: New Poster\r\n\r\nContent"
      end

      should_respond_with 422
      should "not create the item" do
        assert assigns(:item).new_record?
      end
    end

    context "on POST to :create with a valid item" do
      setup do
        MoviePoster.any_instance.expects(:valid?).returns(true)
        post :create, :message => "From: #{@user.email}\r\nSubject: New Task\r\n\r\nContent"
      end

      should_respond_with 201
      should "set the items title to be the message subject" do
        assert_equal "New Task", assigns(:item).title
      end
    end
  end
end

I have also created a really simple email message to pass as the message param here as an example with the header and body separated by two carriage return new lines but this isn’t really necessary with the use of mocha.

From: example@example.com
Subject: Subject

content

So on to the Mailer

Now that we have our controller sorted and fully tested we can create our Mailer, which will automatically create a functional test. It’s then just a case of adding the tests to test the receive method. The first step is to create a sample email that we can use in each of our tests. I am using the Mail gem again to do this.

  context "A MoveMailer" do
    setup do
      @message = Mail.new do
        from "test@example.com"
        to "test@example.com"
        subject "New Movie"
      end
    end

Then we can add our tests making use of the example mail we added and changing any of the parts we want to.

  should "create a new movie poster with the email subject" do
    @message.subject = "new subject"
    assert_difference("Movie.count", 1) do
      movie_poster = MovieMailer.receive(message)
      assert movie_poster.persisted?, movie_poster.errors.full_messages
      assert_equal "new subject", movie_poster.subject
    end
  end

In this case we’re just going to add one test and we can now create our simple movie mailer to take the subject from the email and store it as a poster.

class MovieMailer < ActionMailer::Base

  # Called whenever a message is received on the movies controller or through the rails runner
  def receive(message)
    # For now just take the first attachment and assume there is only one
    attachment = message.attachments.first

    # Create the movie itself
    Movie.create do |movie|
      movie.title = message.subject
    end
  end
end

Now when your app goes into production you just deliver the message to the controller using the HTTP Post approaches, directly to the mailer using any of the other approaches discussed in the previous article.

Although the example is a little contrived you can hopefully See how you can benefit from testing at the development machine to ensure you system works perfectly once you put your code into production. You also have the added benefit of not having to keep checking things whenever you change your code. Just re-run your tests, same as you would with any other part of your app!

Multipart Body – A gem for working with multipart data

Multipart queries are used quite a lot in the transfer of data around the Internet. There are a number of projects out there that will generate multipart content such as email libraries and even web frameworks for uploading and working with files. When we came create parts of CloudMailin we couldn’t find a gem that would easily allow us to encode multipart content the way we wanted to. We could have used a library that already this ability it baked in but most of them didn’t work with eventmachine and if they did then we couldn’t be sure that they would work with any testing tools that we created later that didn’t rely on eventmachine. Although loads of libraries were implementing this code we couldn’t find anything that was standalone that we could just use across any of the different libraries that could post content.

In order to solve this issue we created our own internal multipart creation code. This weekend we have released that code as a gem called multipart_body. This gem is far from perfect and we have a list of things that we don’t have time to add and we would love some help with but the code has been useful to us so we hope it will be useful to others too.

The gem itself consits of two parts. Multipart body and the parts. To get started just install the gem


$ gem install multipart_body

Once the gem is installed you can create a form-data multipart body using this quick hash shorthand.

require 'multipart_body'
multipart = Multipart.new(:field1 => 'content', :field2 => 'something else')

To get a little more control you can create the parts yourself and use them to create the body:

# using a hash
part = Part.new(:name => 'name', :body => 'body', :filename => 'f.txt', :content_type => 'text/plain')

# or just with the name, body and an optional filename
part = Part.new('name', 'content', 'file.txt')
multipart = Multipart.new([part])

You can also pass a file to the multipart hash to automatically assign the filename:

require 'multipart_body'
multipart = Multipart.new(:field1 => 'content', :field2 => File.new('test.txt'))

The resulting output can then be created as follows:

part.to_s #=> The part with headers and content
multipart.to_s #=> The full list of parts joined by boundaries

So the following code example will create the output that follows:

multipart = MultipartBody.new(:test => 'content', :myfile => File.new('test.txt'))
------multipart-boundary-808358
Content-Disposition: form-data; name="myfile"; filename="test.txt"

hello
------multipart-boundary-808358
Content-Disposition: form-data; name="test"

content
------multipart-boundary-808358--

Like I said before the gem is far from perfect. At the moment it doesn’t have any documentation and it is missing quite a few features. By default it assumes you are creating form-data content and encodings are completely missing at the moment.

Hopefully though with a little bit of help it can provide a great starting block for anyone wishing to implement multipart bodies so that each library doesn’t have to re-invent this. If anyone has any time I’d love to see patches to bring this up to something much more useful.

Receiving Incoming Email in Rails 3 – choosing the right approach

When it comes to sending mail in Rails there are plenty of solutions and the best practices are fairly widely known. There are also a number of third party systems that can help you out. Receiving email in Rails however is a slightly less documented prospect.

Here I will cover some of the basic options for receiving email with some of the advantages and disadvantages. In later articles I will cover how to set up some of the solutions including the server. In each case I will also give a small example showing how you can find a user from the message’s to field update another from the message body. I don’t want to get too into the setup specifics of each approach at this point, instead I want to point out the alternatives and how you can make use of each. From what I can tell there are four main alternatives:

  • The ‘Official Way’ – using a mail server and script/rails runner
  • Using a mail server and cURL
  • Polling using IMAP/POP3
  • Use a Service Provider

It should be noted that I am the creator of one of the service providers (CloudMailin) however I appreciate that not all people want to use external services or have different needs and I am trying to make this article as objective as possible. Having said that if you do have comments please feel free to contact me.

Receiving Email the ‘Official Way’

The rails documentation is pretty sparse on incoming emails however the guides do contain a small snippet. Firstly you need to create a receive method in your ActionMailer. Something like the following for our example:

class MyMailer < ActionMailer::Base
  def receive(message)
    for recipient in message.to
      User.find_by_email(recipient).update_attribute(:bio, message.body)
    end
  end
end

As you can see the ActionMailer class is quite simple, then all that is left is to wire up your server so that any incoming email is sent directly to ActionMailer. This can be done by making sure that your mail server executes the following command:

app_dir/script/rails runner 'MyMailer.receive(STDIN.read)'.

This approach has some serious disadvantages though, especially when it comes to scalability. Firstly every time you receive an email you are spawning an new instance of your environment with script/rails. This is a nightmare in itself. Along with this you also need a copy of your app on the same server as the mail server. So you either have to add the mail server to your app server or you need another server and copy of your app running for the mail. You also have the hassle of setting up a dedicated mail server just for the purpose of receiving these incoming emails.

The same approach using cURL

In order to improve this method it is possible to remove the call to script/rails runner and replace it with a call to the web app via HTTP using cURL. Using this method when a new email arrives the following is called:

ruby receiver.rb

Then we create our receiver something like the following:

# note the backticks here execute the command
`curl -d "message=#{STDIN.read}" http://localhost/incoming_messages`

Update: In the comments it turns out that some people have reported problems with this method. You may need to escape the content so that your app receives the message correctly. The following method should help:

require 'cgi'
# note the backticks here execute the command
`curl -d "message=#{CGI.escape(STDIN.read)}" http://localhost/incoming_messages`

You could of course remove Ruby from the mix here entirely but using a Ruby script allows you to perform any processing if you want to in a more complex example. cUrl -d will send the request as application/x-www-form-urlencoded but you could also send the data multipart/form-data if you wish.

You can then simply create a normal controller and use the create method to receive your email as an HTTP POST. Something like the following:

def create
  message = Mail.new(params[:message])
  for recipient in message.to
      User.find_by_email(recipient).update_attribute(:bio, message.body)
    end
  end
end

This method has the advantage of being a little more scalable as nothing really changes in terms of your app. You simply receive the message over HTTP like any other form post or file upload. You may want to opt to move the processing out to a background job though if you are doing anything complex with the message. You will still however need to install and setup your own mail server.

Using a Third Party

In the last example we received email via an HTTP Post as a webhook. There are a couple of options for taking the setup and monitoring stress out of receiving mail in this manor without having to install an configure a mail server. Two of the options here are CloudMailin and smtp2web.

CloudMailin is currently in free beta and allows you to register to receive email via HTTP Post. The system was designed to be scalable and provide some additional features like delivery logs to make sure your app is receiving the emails. That’s enough about that one as I don’t want to be biased.

smtp2web is a google app engine instance that can be used to achieve a similar goal. It make use of app engines ability to receive email and then forwards the messages on to your web app.

Both of these options are designed to operate in ‘the cloud’ and don’t require you to own or setup a mail server to do the work. You will again probably want to make sure that you move processing over to a background worker if you have anything complex to do so that the processing doesn’t take up resource that should be serving your app to your customers.

Polling Using IMAP or SMTP

Finally this solution makes sense when you need to collect messages from an existing mailbox. You don’t have to own your own mail server but you will need to be able to run cron or a daemon to collect mail at regular intervals.

Although you could roll your own collector there are a couple already out there. Take a look at mailman for example. This approach can either rely on direct acces to your blog or can again POST via HTTP.

I will also look to write a separate post on MailMan as I think the power offered by MailMan is a worth a blog post in itself. Although there will be a delay with any polling as you can only poll every few minutes, in some situations using an existing mailbox is the only option.

Although this was brief, it should have given a quick introduction into some of the approaches available (I’m sure there are more too). I also plan to write a number of follow up articles showing how to implement options described here. If you have any advice, an alternative option or even an approach you would prefer to see covered first then please jump in and comment. Again if you have any comments on CloudMailin please let me know on here, twitter or via email at blog-comments [you know what goes here] cloudmailin.com

Render ‘Rails Style’ Partials in Sinatra

We love Sinatra. Not only does it make a great framework in its own right but in addition it can be used to mimic parts of rails in a real simple environment for front-end designers. Instead of having to get them set up and explain the whole of rails they just get a nice simple app to work on without having to worry about creating different controllers or even models.

Although there is not a 1 to 1 translation between a rails app and a sinatra one, it does allow these developers to work with things like haml in a really easy to work with environment.

One of the features that I was asked for recently though was “How do you render a partial in sinatra?”

Rendering Partials in Sinatra

Sinatra is a super-lightweight framework. Because of this it doesn’t have the notion of partials built into it. However, a partial, in its simplest form, is nothing more than a call out to render the template as a string and then embed that string into your page.

A quick look at the sinatra sites FAQs shows that partials can be rendered in the following way in erb.

<%= erb(:mypartial, :layout => false) %>

In haml you could use exactly the same thing but call haml like so.

= haml(:mypartial, :layout => false)

Notice that

:layout => false

is set to ensure that the layout is not also rendered.

Going a little further

The FAQs also recommend using the code in the following gist.

http://gist.github.com/119874

The code shows a helper method called partial. This helper method can be used to render a partial from your code. The helper also allows you to pass collections and is a really cool and useful piece of code.

Making things work the rails way

The above helpers are great and really useful for sinatra. However, what if you want to render a partial the ‘rails way’? In our situation we were using sinatra as a mock up of what would eventually be brought into a rails app. Rails allows partials to be included like so:

<%= render :partial => 'partial_name' %>

By overriding the built in render method in Sinatra it is actually possible to mimic the rails partials. I came up with the following helper to quickly mock things up. The helper checks to see if the first argument passed to is a hash and if that contains they key :partial. If so it renders the partial, if not it just uses the default render method.

  helpers do
    def render(*args)
      if args.first.is_a?(Hash) && args.first.keys.include?(:partial)
        return haml "_#{args.first[:partial]}".to_sym, :layout => false
      else
        super
      end
    end
  end

The helper could easily by extended to allow for collections etc but for now it does the job. Any better solutions?