Remote controlled Home Automation using Sinatra, Ruby and the LightwaveRF Wifi box

In the last post I created a Ruby Gem to communicate with the LightwaveRF Wifi Link Box. The next step was getting that gem into a web server so that I could send commands to the LightwaveRF box over HTTP. The eventual aim was to get this deployed on a RaspberryPi so that it could act as an always-on webserver to control any of the devices in the house.

The obvious choice for a lightweight and simple webserver in Ruby is Sinatra. The following code was all it took to get things started:

class App < Sinatra::Base
  def initialize
    super
    @lightwave = LightwaveRF.new
  end

  get "/" do
    body "Usage: turn items on or off: /[on|off]/:room/:device to dim /dim/:room/:device/:level. Level should be between 0 and 100."
  end

  get "/:room/:device/:action/?:level?" do |room, device, action, level|
    return [422, "Level required for dim"] if action == "dim" && !level
    case action
    when /on/i
      @lightwave.turn_on(room, device)
    when /off/i
      @lightwave.turn_off(room, device)
    when /dim/i
      puts level
      @lightwave.dim(room, device, level.to_i)
    else
      return [422, "Unknown action #{action}"]
    end
    body "Room #{room} Device #{device} #{action}#{" to #{level}%" if level}."
  end
end

The code starts off by creating an instance of the LightwaveRF gem. Then you can just call the root page on the server to see the interface. Then you can use the following to turn on room 1, device 2.


http://localhost:9292/on/1/2

or to Dim room 2, device 3 to 50% you could use:


http://localhost:9292/dim/2/3/50

Don’t forget you’ll need to register the device first by calling LightwaveRF.new.register from the computer that’s running the webserver.

If you’re not familiar with Ruby the following should help you get setup in the terminal assuming that you have ruby installed:

git clone git@github.com:scsmith/lightwaverf-sinatra.git
cd lightwaverf-sinatra
bundle install --path ./vendor/bundle
bundle exec ruby -e "require 'lightwave_rf'; LightwaveRF.new.register; puts 'Done'"
bundle exec rackup -p 9292

This will register the device and start the server running on port 9292 (the default for sinatra). You can then call http://localhost:9292/ like above to start using the server.

Home automation with Ruby using the LightwaveRF Wifi Link box

I recently got to play with the LightwaveRF Wifi Link box (http://www.lightwaverf.com/). They’re becoming fairly common in the UK with companies like screwfix selling them. You can get a range of devices to pair with the box including lights, plug sockets and even PIR sensors. We plugged things in, got a lightbulb paired up and had a play. It’s pretty cool however the next step was writing our own software to control it. Who doesn’t want to control their house using Ruby?

Unfortunately the LightwaveRF Wifi Link doesn’t come with an API or any obvious way to communicate with it outside the tools that the company provide. I was just about ready to get out wireshark and try to figure out the protocol it was using to communicate however, fortunately someone had already done this.

It turned out talking to the box is pretty simple over UDP. First you have to initialise the conversation and pair the device you want to use to communicate with the Wifi box. You can do that with the following:

socket = UDPSocket.new
socket.setsockopt(Socket::SOL_SOCKET, Socket::SO_BROADCAST, true)
socket.send("533,!R1D1F0|", 0, "255.255.255.255", 9760)
socket.close

Once you’ve done this commands are sent in the following format:

"001,!R1D2F1"

Prefix: 001
Room: 1
Device: 2
Function Code: F1

The prefix is a random number between 001 and 099 Function code F0 means turn off, F1 means turn on and to Dim a device you can send FdPXX where XX is a number between 1 and 32, with FdP32 meaning full brightness. So the following code would dim Room 2, Device 3 to 50% brightness:

socket = UDPSocket.new
socket.setsockopt(Socket::SOL_SOCKET, Socket::SO_BROADCAST, true)
socket.send("001,!R2D3FdP16|", 0, "255.255.255.255", 9760)
socket.close

To make this easier I’ve published the LightwaveRF Ruby Gem to let you interact with the device from Ruby in a much easier way.

lightwave = LightwaveRF.new

# Initial Registration (Only needs doing once per device)
lightwave.register

# Turn the light on in room 1, device number 1
lightwave.turn_on(1, 1)

# Dim the lights in room 2, device number 3 after turning them on
lightwave.turn_on(2, 3)
lightwave.dim(2, 3, 50)

# Turn off the lights
lightwave.turn_off(1, 1)
lightwave.turn_off(2, 3)

For more details or if you find any other features to add checkout the project on Github. The next step was to add this gem to a small Sinatra webserver, which I’ll post soon.

Converting a github issue into a pull request

Converting a github issue into a pull request

Recently someone submitted an issue for one of my open source github repositories (https://github.com/scsmith/language_list). Language list imports a pretty huge list of lanauges as it loads and this was taking a bit of time. There was quite a bit of discussion on the issue, with a fair few branches. Eventaully we were happy with the changes in one of the branches and wanted to bring them into master.

There were two options here, the first is to create a new pull request and just reference the existing issue (you can do this by typing # and using the autocomplete). However I also managed to find this stackoverflow question and answer giving a more preferable option. You can convert an existing issue into a pull request itself. You need to use the github api though but this is pretty easy using curl:

curl --user "your_username:your_password" --request POST --data '{"issue": "4", "head": "username:reponame", "base": "master"}' https://api.github.com/repos/username/reponame/pulls

In this head is the repo you want to bring in and base is the subject of the pull that you want to merge to.

There’s also a site that can do this for you http://issue2pr.herokuapp.com/ but you will have to give full access to your private repos. Personally I find the curl approach pretty simple and really liked this option. Remember to head over to the stackoverflow question and upvote the answer if you use it!

Installing Rails 3.2 on ArchLinux Raspberry Pi

Before I was playing with Debian on the Raspberry Pi I was messing with ArchLinux and managed to get Rails 3.2 running on the Pi. I’m posting this up as a quick guide for anyone else that might want it.

The first step is to log into the command prompt either directly or via ssh. I’m not going to add a different user at this point just set things up for the root user but you might want to consider doing this differently for anything but a toy project.

Next we need to install the dependancies. Arch uses a dependancy manager called pacman. Let’s start by updating pacman itself to make sure it’s up to date.

pacman -S pacman

We then need to install the rest of the dependancies:

pacman -S ruby # ruby itself.
pacman -S base-devel # make etc for compiling gems
pacman -S libxml2 # used by gems like nokogiri for xml
pacman -S libxslt # used by gems like nokogiri
pacman -S sqlite # sqlite database
pacman -S nodejs # Rails 3.2 needs a javascript runtime. Node is one of the options.

I wanted to run solr for search so I also installed openJDK6 but I suspect most projects won’t need this.

pacman -S openjdk6

With the dependancies installed we should now have ruby 1.9 installed. We can just check this using the version flag on ruby.

ruby -v

Now we just need to install bundler:

gem install bundler

With bundler installed we’ll need to copy the code over to the device (or generate a new rails app by installing the rails gem and running the generator). You could easily do this yourself using scp or a similar tool but I used capistrano here with the following recipe:

set :application, "manifest"
set :repository,  "."
set :deploy_via, :copy
set :scm, :git

set :user, "root"
set :password, "root"
set :use_sudo, false

role :web, "192.168.1.190"
role :app, "192.168.1.190"

Rather than checking out the git repo remotely I’m just asking capistrano to copy my local project which uses git and providing it the Raspberry Pi’s username, password and IP address.

Once the code is deployed simply run bundler to install the gems.

bundle install

Once the gems are installed go ahead and migrate up the database:

rake db:schema:load

and Start Rails!

script/rails server -p 80

You should now be able to access your rails app over the network!. There’s nothing groundbreaking in here but I thought I’d document this in case it helps anyone.

Controlling access to Routes and Rack apps in Rails 3 with Devise and Warden

Devise is a great authentication solution for use within your Rails apps. We’ve used it in a number of projects and have always been happy with the flexibility but never realised quite how great it was until very recently.

Recently we wanted to make use of an A/B testing solution and one of the options was Split. Like a number of projects nowadays (Resque included) Split comes with an embedded server that allows you to control it via the web interface.

You can mount that rack app using:

mount Split::Dashboard, :at => 'split'

The problem is this leaves split open for anyone to see and your AB testing is open for anyone to play with. The split guide recommends that you can use a standard rack approach to protect your split testing.

Split::Dashboard.use Rack::Auth::Basic do |username, password|
  username == 'admin' && password == 'p4s5w0rd'
end

We were already using Devise though and I’m sure this is only there to prove as an example by the authors. What we really wanted to do was use Devise to protect our mounted app. It turns out that this isn’t that difficult. Firstly you can authenticate any route using devise using the authenticated command like the following:

authenticated(:user) do
  resources :post
end

This ensures that the following route, specified with the block, is only availible to people who have authenticated. Using authenticate you can also bounce people to the login page if they’re not authenticated:

authenticate(:user) do
  resources :post
end

This is awesome and ensures that only registered users can access the routes. You can of course replace :user with any of your scopes, such as :admin, in order to ensure that only specific scopes have access to the route. However, we wanted a bit more control. What if you want to only allow any users that had the boolean `allowed_to_ab_test?` method?

As soon as you realise that:

mount Split::Dashboard, :at => 'split'

is actually just a short hand for

match "/split" => Split::Dashboard, :anchor => false

You can use any normal constraint. Since warden makes use of the request object you can pull out any details you like from warden including the user model by using a lambda for the constraint. This provides massive flexibility for controlling your routes:

match "/split" => Split::Dashboard, :anchor => false, :constraints => lambda { |request|
  request.env['warden'].authenticated? # are we authenticated?
  request.env['warden'].authenticate! # authenticate if not already
  # or even check any other condition
  request.env['warden'].user.allowed_to_ab_test?
}

With that example we can take a really fine grained control of our routes and only allow access to certain routes for certain people without having to specify different scopes for the users.

Nothing here’s groundbreaking but it took us a little time to figure it out. The combination of Devise and Warden gives a huge amount of power and flexibility allowing you to protect both your normal routes or your embedded apps. We thought we’d share this in case it was useful to others. It’s certainly opened our eyes to just how great Devise and Warden are!

Display error_messages_for for a single flash error using a helper

So error_messages_for has been deprecated in Rails 3. However, you can still download the plugin from here and it’s still in use in a number of projects that we come across. It’s quite often that you see flash error messages appear outside of this error messages for style box but what if you just wanted to display a standardised form across your page.
In it’s simplest form the standard error_messages_for accepts an active model object and will display all of the errors found on that object. But what if we want to just pass in the error message that it should display?

The following snippet allows you to make use of the same template used by default but accepts a string containing the error message.

  def error_from_message(error_message, options = {})
    return nil unless error_message
    
    header_message = options[:header_message] || t(:header, :count => 1, :scope => [:activerecord, :errors, :template], :model => 'item')
    message = options[:message] ||  t(:body, :scope => [:activerecord, :errors, :template])
    
    contents = ''
    contents << content_tag(options[:header_tag] || :h2, header_message)
    contents << content_tag(:p, message)
    contents << content_tag(:ul, content_tag(:li, error_message))
    content_tag(:div, contents.html_safe, :id => 'errorExplanation', :class => 'errorExplanation')
  end

If you place that into your application helper you can then call the following in the view to render the standard active model style error message for the string you supply:

<%= error_from_message flash[:error_message] %>

You can also pass some of the standard parameters to change the tag or override the default header message. There’s nothing amazing in this but hopefully it will at least show where to get started to access the standard i18n translations used by the Rails helpers.

Language List Gem – a list of languages (ISO-639-1 or ISO-639-3) for Ruby

For a project I had been working on we needed to allow users to select the Languages that they could speak as part of a profile page. After a bit of research I found it difficult to find a comprehensive list of languages available in a format that could easily be used by Ruby.

Rails’ build in language files means that we’re all used to these shorthand prefixes like en to mean English. The issue was finding a nice list of all possible languages that could resolve to these codes. If we don’t take localised dialects into account such as en-gb or en-us then we can make use of the ISO standard list of languages housed in ISO-639-1 and ISO-639-3 for example.

After creating a script to compile these languages into a yml file that could easily be retrieved I launched this as a Ruby Gem that can be used in the following way:

all_languages = LanguageList::ALL_LANGUAGES

# Finding a language based on its ISO-639-1 or ISO-639-3 code
english = LanguageList::LanguageInfo.find('en')
english.name.inspect #=> "English"
english.iso_639_1.inspect #=> "en"
english.iso_639_3.inspect #=> "eng"

The only thing left was to add some helpers to be able to retrieve a list of common languages. Rather than re-invent the wheel I found a list of common langauges on this website. The author has compiled the list of common languages based upon the list of langauges that Microsoft use and a Wikipedia page of common languages.

I then added a common? flag to these langauges allowing for the following:

all_languages = LanguageList::ALL_LANGUAGES
common_languages = LanguageList::COMMON_LANGUAGES

# Finding a language based on its ISO-639-1 or ISO-639-3 code
english = LanguageList::LanguageInfo.find('en')
english.name.inspect #> "English"
english.iso_639_1.inspect #=> "en"
english.iso_639_3.inspect #=> "eng"
english.common? #=> true

I’m not 100% certain of the license of the ISO country list so use it at your own risk but any of the code I have created is released under the MIT license.

I am also sure that the datafile itself could be tweaked to include better names and punctuation so send any pull requests to the project on GitHub.

Receiving and Saving Incoming Email Images and Attachments with Paperclip and Rails 3

The last post showed how the use of Test Driven Development can help create Rails apps that can receive email offline that can be plugged into one of the incoming email solutions to receive email. The example was a movie poster model that took the subject of the email and added it as the title of that movie. The thing is what’s a movie poster without images? In this post I am going to cover how you can extract an image out of an email and attach it to a model using Paperclip.

We’ve already covered how to receive the email using CloudMailin and the app’s controller and pass it onto a mailer so I’m only going to cover the code we need in our mailer and models here.

Getting Paperclip setup

The first step is to get paperclip installed into our app. I would recommend following the instructions in the Paperclip readme here to get your app set up with Paperclip and test everything works with a normal controller and form. Our movie poster will need the following setup:

We start by adding the migration to add the Paperclip columns to the migration

t.string    :poster_file_name
t.string    :poster_content_type
t.integer   :poster_file_size
t.datetime  :poster_updated_at

We also need to add the magic Paperclip line to our Movie model

has_attached_file :poster, :styles => { :medium => "300x300>", :thumb => "100x100>" }

So far everything has been exactly the same as we would normally have to do if we were using the normal forms based approach to upload, next we’ll add the code to our mailer.

Setting up the Mailer to Receive Attachments

Normally when Rails receives an uploaded file it has to be through a multipart/form-data. When a Rails app receives these files it recognises the attachment parameter and behind the scenes a temporary file is created containing that content. This is to stop the attachment being stored in memory. This file is then extended by Paperclip to contain a few additional bits of information it’s original file name and the content type of the upload. Our email however is stored in memory and it’s attachments don’t get stored as files. For that reason we need to emulate the temporary file so that Paperclip can correctly work with out attachments.

In order to do this we can use the following mailer:

class MovieMailer < ActionMailer::Base
  # Create an attachment file with some paperclip aware features
  class AttachmentFile < Tempfile
    attr_accessor :original_filename, :content_type
  end

  # Called whenever a message is received on the movies controller
  def receive(message)
    # For now just take the first attachment and assume there is only one
    attachment = message.attachments.first

    # Create the movie itself
    Movie.create do |movie|
      movie.title = message.subject

      # Create an AttachmentFile subclass of a tempfile with paperclip aware features and add it
      poster_file = AttachmentFile.new('test.jpg')
      poster_file.write attachment.decoded
      poster_file.flush
      poster_file.original_filename = attachment.filename
      poster_file.content_type = attachment.mime_type
      movie.poster = poster_file
    end
  end
end

Here we can see the AttachmentFile class that fakes the parameters needed by Paperclip to work with the file. We then create the AttachmentFile class passing in a filename to be used for the temporary file. Then we write to the file, flush the contents and set the required parameters for Paperclip.

Now when we receive an email the first attachment is taken from the email and stored as our movie poster with by Paperclip.

If you don’t want to use a temporary file you can also use a StringIO object and see the same benefits.

In addition to this @pedrodelgallego pointed out another posible solution that extends a StingIO class anonymously using class eval using the following code:

attachment = message.attachments.first

file = StringIO.new(attachment.decoded)
file.class.class_eval { attr_accessor :original_filename, :content_type }
file.original_filename = attachment.filename
file.content_type = attachment.mime_type

Hopefully this will be of use for anyone trying to store attachments extracted from emails. The real benefit is that Paperclip doesn’t only work with images, you can store all sorts of attachments and push them to S3 if you need to. If anyone has any other suggestions or better ways to achieve this I’d also love to hear them.

Receiving Test Driven Incoming Email for Rails 3

One of the most frequent questions we are asked at CloudMailin is how can you develop your application and receives email offline, without opening ports, in a Rails app.

There are options that spring to mind. The first is to write a script that delivers a local email to your app simulating the response that you would get from an email server. The second solution is far simpler. You can perform test driven development and it can be really easy  get things up and running in development and even easier to move to production.

Here we are going to create a really simple example to take the subject of an email and create a model called movie poster from that subject. In a later post we will show how we can also use the attachments to create the images for the poster.

The First Step, a Controller to Receive the Email

In a previous article I highlighted a number of options for Receiving Email in Rails 3. Using some of the approaches you don’t even need to use a controller but in that case where email is delivered to your app via an HTTP POST the controller is needed. The controller doesn’t have to be complicated though. All we really want to do is pass a mail object to the receive method of our mailer. So supposing we have a mailer called MovieMailer we can do the following:

class IncomingMailsController < ApplicationController
   skip_before_filter :verify_authenticity_token
   
   def create
     # create a Mail object from the raw message
     movie_poster = MovieMailer.receiver(Mail.new(params[:message]))
     # if the movie poster was saved then send a status of 201 else give a 422 with the errors
     if !movie_poster.new_record?
        render :text => "Success", :status => 201, :content_type => Mime::TEXT.to_s
    else
      render :text => movie_poster.errors.full_messages.join(', '), :status => 422, :content_type => Mime::TEXT.to_s
   end
  end
end

The example above assumes that you are using at least Rails 3.0 but it can easily be converted to work with older versions of Rails and you could use the TMail gem rather than Mail if you need to.

Now we have this controller we can create a functional test to check this in the same way we would any other controller in our app (note I’m doing this the wrong way round for illustration you should really write your test first!). If we mock the response from the Mailer for now we can use something like this with Shoulda but you could use RSpec or any other testing framework just as easily:

class IncomingMailsControllerTest < ActionController::TestCase
   context "An IncomingMailsController" do
     context "on POST to :create with an invalid item" do
        setup do
           MoviePoster.any_instance.expects(:valid?).returns(false)
           post :create, :message => "From: #{@user.email}\r\nSubject: New Poster\r\n\r\nContent"
      end

      should_respond_with 422
      should "not create the item" do
        assert assigns(:item).new_record?
      end
    end

    context "on POST to :create with a valid item" do
      setup do
        MoviePoster.any_instance.expects(:valid?).returns(true)
        post :create, :message => "From: #{@user.email}\r\nSubject: New Task\r\n\r\nContent"
      end

      should_respond_with 201
      should "set the items title to be the message subject" do
        assert_equal "New Task", assigns(:item).title
      end
    end
  end
end

I have also created a really simple email message to pass as the message param here as an example with the header and body separated by two carriage return new lines but this isn’t really necessary with the use of mocha.

From: example@example.com
Subject: Subject

content

So on to the Mailer

Now that we have our controller sorted and fully tested we can create our Mailer, which will automatically create a functional test. It’s then just a case of adding the tests to test the receive method. The first step is to create a sample email that we can use in each of our tests. I am using the Mail gem again to do this.

  context "A MoveMailer" do
    setup do
      @message = Mail.new do
        from "test@example.com"
        to "test@example.com"
        subject "New Movie"
      end
    end

Then we can add our tests making use of the example mail we added and changing any of the parts we want to.

  should "create a new movie poster with the email subject" do
    @message.subject = "new subject"
    assert_difference("Movie.count", 1) do
      movie_poster = MovieMailer.receive(message)
      assert movie_poster.persisted?, movie_poster.errors.full_messages
      assert_equal "new subject", movie_poster.subject
    end
  end

In this case we’re just going to add one test and we can now create our simple movie mailer to take the subject from the email and store it as a poster.

class MovieMailer < ActionMailer::Base

  # Called whenever a message is received on the movies controller or through the rails runner
  def receive(message)
    # For now just take the first attachment and assume there is only one
    attachment = message.attachments.first

    # Create the movie itself
    Movie.create do |movie|
      movie.title = message.subject
    end
  end
end

Now when your app goes into production you just deliver the message to the controller using the HTTP Post approaches, directly to the mailer using any of the other approaches discussed in the previous article.

Although the example is a little contrived you can hopefully See how you can benefit from testing at the development machine to ensure you system works perfectly once you put your code into production. You also have the added benefit of not having to keep checking things whenever you change your code. Just re-run your tests, same as you would with any other part of your app!

Multipart Body – A gem for working with multipart data

Multipart queries are used quite a lot in the transfer of data around the Internet. There are a number of projects out there that will generate multipart content such as email libraries and even web frameworks for uploading and working with files. When we came create parts of CloudMailin we couldn’t find a gem that would easily allow us to encode multipart content the way we wanted to. We could have used a library that already this ability it baked in but most of them didn’t work with eventmachine and if they did then we couldn’t be sure that they would work with any testing tools that we created later that didn’t rely on eventmachine. Although loads of libraries were implementing this code we couldn’t find anything that was standalone that we could just use across any of the different libraries that could post content.

In order to solve this issue we created our own internal multipart creation code. This weekend we have released that code as a gem called multipart_body. This gem is far from perfect and we have a list of things that we don’t have time to add and we would love some help with but the code has been useful to us so we hope it will be useful to others too.

The gem itself consits of two parts. Multipart body and the parts. To get started just install the gem


$ gem install multipart_body

Once the gem is installed you can create a form-data multipart body using this quick hash shorthand.

require 'multipart_body'
multipart = Multipart.new(:field1 => 'content', :field2 => 'something else')

To get a little more control you can create the parts yourself and use them to create the body:

# using a hash
part = Part.new(:name => 'name', :body => 'body', :filename => 'f.txt', :content_type => 'text/plain')

# or just with the name, body and an optional filename
part = Part.new('name', 'content', 'file.txt')
multipart = Multipart.new([part])

You can also pass a file to the multipart hash to automatically assign the filename:

require 'multipart_body'
multipart = Multipart.new(:field1 => 'content', :field2 => File.new('test.txt'))

The resulting output can then be created as follows:

part.to_s #=> The part with headers and content
multipart.to_s #=> The full list of parts joined by boundaries

So the following code example will create the output that follows:

multipart = MultipartBody.new(:test => 'content', :myfile => File.new('test.txt'))
------multipart-boundary-808358
Content-Disposition: form-data; name="myfile"; filename="test.txt"

hello
------multipart-boundary-808358
Content-Disposition: form-data; name="test"

content
------multipart-boundary-808358--

Like I said before the gem is far from perfect. At the moment it doesn’t have any documentation and it is missing quite a few features. By default it assumes you are creating form-data content and encodings are completely missing at the moment.

Hopefully though with a little bit of help it can provide a great starting block for anyone wishing to implement multipart bodies so that each library doesn’t have to re-invent this. If anyone has any time I’d love to see patches to bring this up to something much more useful.

Follow

Get every new post delivered to your Inbox.

Join 362 other followers