A Step by Step Guide to Receiving Email in your Web Application with CloudMailin

In this post, we’re going to take a step by step guide approach to configuring and receiving your first email in you web app with CloudMailin. CloudMailin allows you to receive email in your web app via an HTTP POST. You setup the server to point to your site’s public url and your email is sent straight to the site. We will also cover how to add your own domain name and set it up so that you can give customers your own email address.

The first step is to head to http://CloudMailin.com and signup for a free account.

As soon as you sign up, you will be presented with the option to create your CloudMailin address. Here you enter the url of page that you wish to receive your email via HTTP Post at. After you click submit, an email address will be generated for you and anything you send to that email address will be sent to your website.

Once you click the submit button, the site will generate your email address and you will be ready to go. You will be redirected to the address list page. Here you can see each of your CloudMailin addresses, the target, that your HTTP Post will be sent to when an email is received and the plan details. Each CloudMailin address sits in its own plan so you can have different allocations for each address and therefore web app you want to receive email within. Currently the site is in beta so there is only one free plan. However once the site launches, there should always be a free plan to get started.

If you click on the address listed in the address list, you are taken to the page for that address. On this page are a number of configuration options for the address but also the delivery status list. The delivery status list is a powerful feature that allows you to see each email that passes through the system and the status of that message.

So lets go ahead and send an email to the address that CloudMailin has given us. We should almost instantly get a response back saying that the message delivery has failed.

What? The delivery failed? When we take a look at the delivery status page we can not only see that the message has failed but see that the target gave a status code of 404 because we never set up a page to receive the email. CloudMailin is clever enough to know the difference between the status codes that it receives when it delivers the email. If it receives a status code like 200 it assumes that everything was ok and the message was received successfully. If the target server has an error then or is unavailable then CloudMailin will tell the server that an error has occurred and that it should retry later. Finally if a status code like 404 or 422 occurs then it will tell the server that the message was rejected and that it should not try again later. More details about the HTTP responses that you can give to CloudMailin and the actions taken can be found in the HTTP status codes documentation. You can even send a custom bounce message when you reject the delivery of an email so long as your response is in plain text.

Ok so now its time to write some code and receive our email. If you are using Rails, you can take a look at some previous blog posts. There are also some examples in the parsing the email documentation. CloudMailin is language agnostic though, you can use it to receive incoming email in Java, PHP, Python, .Net, Scala, Small Talk or any other language that you want (so long as it can parse HTTP Posts). We are hoping to expand the languages in the documentation and they are all on github so feel free to contribute documentation for your favorite language! The request is sent to your server in the same way as any other form that you would fill in on your site as a multipart/form-data request. The plaintext and html parts are also nicely separated out to make life easier but the full email is available to allow more detail and attachments if it’s needed. The HTTP Post format documentation should help here.

Great, so now we have our server configured correctly, we’re ready to receive our email. So lets go ahead and send another email to our CloudMailin email address.

Awesome, Green lights! So now we’re successfully receiving our emails and processing the contents. What else can we do? Well, if you take a look down the right hand side of our address page, there’s a button labeled custom domains. This allows us to use our own domain name to receive email with whatever address we want. In order to do this though we need to start by configuring our DNS server to point any emails sent to our domain to the CloudMailin server. The documentation has a page dedicated to showing you how to configure your DNS server’s MX records to point to the CloudMailin servers and allow you to receive email via your own domain.

Once we have configured our DNS records we can go ahead and add our domain name to the custom domains form that we can open using the custom domains button. If you want to create client subdomains like anything@x.example.com and anything@y.example.com you can also add subdomains.

So that’s it. Using CloudMailin we can pretty easily receive email in our web apps. We’d love to hear what you create and also remember all of the documentation is on Github so if you have changes or additions or importantly examples for other programming languages and frameworks then please fork the documents and send a pull request. Most importantly feedback is always appreciated.

Receiving Incoming Email in Rails 3 – choosing the right approach

When it comes to sending mail in Rails there are plenty of solutions and the best practices are fairly widely known. There are also a number of third party systems that can help you out. Receiving email in Rails however is a slightly less documented prospect.

Here I will cover some of the basic options for receiving email with some of the advantages and disadvantages. In later articles I will cover how to set up some of the solutions including the server. In each case I will also give a small example showing how you can find a user from the message’s to field update another from the message body. I don’t want to get too into the setup specifics of each approach at this point, instead I want to point out the alternatives and how you can make use of each. From what I can tell there are four main alternatives:

  • The ‘Official Way’ – using a mail server and script/rails runner
  • Using a mail server and cURL
  • Polling using IMAP/POP3
  • Use a Service Provider

It should be noted that I am the creator of one of the service providers (CloudMailin) however I appreciate that not all people want to use external services or have different needs and I am trying to make this article as objective as possible. Having said that if you do have comments please feel free to contact me.

Receiving Email the ‘Official Way’

The rails documentation is pretty sparse on incoming emails however the guides do contain a small snippet. Firstly you need to create a receive method in your ActionMailer. Something like the following for our example:

class MyMailer < ActionMailer::Base
  def receive(message)
    for recipient in message.to
      User.find_by_email(recipient).update_attribute(:bio, message.body)
    end
  end
end

As you can see the ActionMailer class is quite simple, then all that is left is to wire up your server so that any incoming email is sent directly to ActionMailer. This can be done by making sure that your mail server executes the following command:

app_dir/script/rails runner 'MyMailer.receive(STDIN.read)'.

This approach has some serious disadvantages though, especially when it comes to scalability. Firstly every time you receive an email you are spawning an new instance of your environment with script/rails. This is a nightmare in itself. Along with this you also need a copy of your app on the same server as the mail server. So you either have to add the mail server to your app server or you need another server and copy of your app running for the mail. You also have the hassle of setting up a dedicated mail server just for the purpose of receiving these incoming emails.

The same approach using cURL

In order to improve this method it is possible to remove the call to script/rails runner and replace it with a call to the web app via HTTP using cURL. Using this method when a new email arrives the following is called:

ruby receiver.rb

Then we create our receiver something like the following:

# note the backticks here execute the command
`curl -d "message=#{STDIN.read}" http://localhost/incoming_messages`

Update: In the comments it turns out that some people have reported problems with this method. You may need to escape the content so that your app receives the message correctly. The following method should help:

require 'cgi'
# note the backticks here execute the command
`curl -d "message=#{CGI.escape(STDIN.read)}" http://localhost/incoming_messages`

You could of course remove Ruby from the mix here entirely but using a Ruby script allows you to perform any processing if you want to in a more complex example. cUrl -d will send the request as application/x-www-form-urlencoded but you could also send the data multipart/form-data if you wish.

You can then simply create a normal controller and use the create method to receive your email as an HTTP POST. Something like the following:

def create
  message = Mail.new(params[:message])
  for recipient in message.to
      User.find_by_email(recipient).update_attribute(:bio, message.body)
    end
  end
end

This method has the advantage of being a little more scalable as nothing really changes in terms of your app. You simply receive the message over HTTP like any other form post or file upload. You may want to opt to move the processing out to a background job though if you are doing anything complex with the message. You will still however need to install and setup your own mail server.

Using a Third Party

In the last example we received email via an HTTP Post as a webhook. There are a couple of options for taking the setup and monitoring stress out of receiving mail in this manor without having to install an configure a mail server. Two of the options here are CloudMailin and smtp2web.

CloudMailin is currently in free beta and allows you to register to receive email via HTTP Post. The system was designed to be scalable and provide some additional features like delivery logs to make sure your app is receiving the emails. That’s enough about that one as I don’t want to be biased.

smtp2web is a google app engine instance that can be used to achieve a similar goal. It make use of app engines ability to receive email and then forwards the messages on to your web app.

Both of these options are designed to operate in ‘the cloud’ and don’t require you to own or setup a mail server to do the work. You will again probably want to make sure that you move processing over to a background worker if you have anything complex to do so that the processing doesn’t take up resource that should be serving your app to your customers.

Polling Using IMAP or SMTP

Finally this solution makes sense when you need to collect messages from an existing mailbox. You don’t have to own your own mail server but you will need to be able to run cron or a daemon to collect mail at regular intervals.

Although you could roll your own collector there are a couple already out there. Take a look at mailman for example. This approach can either rely on direct acces to your blog or can again POST via HTTP.

I will also look to write a separate post on MailMan as I think the power offered by MailMan is a worth a blog post in itself. Although there will be a delay with any polling as you can only poll every few minutes, in some situations using an existing mailbox is the only option.

Although this was brief, it should have given a quick introduction into some of the approaches available (I’m sure there are more too). I also plan to write a number of follow up articles showing how to implement options described here. If you have any advice, an alternative option or even an approach you would prefer to see covered first then please jump in and comment. Again if you have any comments on CloudMailin please let me know on here, twitter or via email at blog-comments [you know what goes here] cloudmailin.com

Fragment Cache Expiration in Ruby on Rails

A few days ago I was trying to implement fragment caching with a timed expiration using memcached in rails. I needed a system to cache the results of querying a large number of rss feeds that were being aggregated on the page. Whilst I could have created a system to do this offline and store the results it was easier to just fragment cache and show a stale cache whilst new results were found.

When manually caching or in action caching you simply pass :expires_in => 5.minutes to tell memcached to automatically expire the page after 5 minutes. However this didn’t seem to work with fragment caching, after a tiny bit of digging and error I realised that in order to specify the options hash you have to first specify a key. Since the key is generally made up of the request url I simply used this and appended a unique id for the fragment. Obvious when you think about it I guess!

cache("#{request.url}?fragment", {:expires_in => 60.seconds}) do
  something
end