If you need to perform any time consuming work in your Rails actions, you’ll probably want to offload this into a background job. There are many different frameworks to help with this, and the one we use is Workling. The nice thing about Workling is that it provides an abstraction layer that allows you to decouple your actual background job implementation from the background execution strategy. For example, in our development environment we are using the Spawn runner (which simply forks the Rails process for each background job), but we need a proper, queue based runner in production. Up until recently we were using the Starling runner, which worked pretty well for a small set of machines.

However, after migrating our infrastructure to Amazon EC2 and rapidly scaling up the number of app servers, we figured it would be great to take advantage of Amazon SQS (Simple Queue Service), rather than maintaining our own queue servers. Fortunately, Workling’s plugin architecture makes it very easy to implement your own clients, so writing an SQS Workling client turned out to be fairly straightforward.

If you are interested in using this in your own project, simply use my Workling fork on Github. I haven’t decided yet whether to extract this into a separate plugin that you could install alongside Workling, so let me know if you have a strong preference. I’ll also get in touch with the Workling developers to see if they might be interested in pulling this feature into the main code base. But for now, you can simply install it by following the regular Workling plugin installation instructions, except using my Workling fork:

script/plugin install git://github.com/digitalhobbit/workling.git

The README includes detailed instructions on configuring the client, but it’s actually very easy:

Install the RightAws gem:

sudo gem install right_aws

Configure Workling to use the SqsClient. Add this to your environment:

Workling::Remote.dispatcher = Workling::Remote::Runners::ClientRunner.new
Workling::Remote.dispatcher.client = Workling::Clients::SqsClient.new

Add your AWS key id and secret key to workling.yml:

production:
  sqs_options:
    aws_access_key_id: <your AWS access key id>
    aws_secret_access_key: <your AWS secret access key>

You can optionally override the following settings, although the defaults will likely be sufficient:

    # Queue names consist of an optional prefix, followed by the environment
    # and the name of the key.
    prefix: foo_

    # The number of SQS messages to retrieve at once. The maximum and default
    # value is 10.
    messages_per_req: 10

    # The SQS visibility timeout for retrieved messages. Defaults to 30 seconds.
    visibility_timeout: 30

    # The number of seconds to reserve for deleting a message from the qeueue.
    # If buffered messages are getting too close to the visibility timeout,
    # we drop them so they will get picked up the next time a worker retrieves
    # messages, in order to avoid duplicate processing.
    visibility_reserve: 10
        
    # Below are various retry and timeout settings for the underlying right_aws
    # and right_http_connection libraries. You may want to tweak these based on
    # your workling usage. I recommend fairly low values, as large values can
    # cause your Rails actions to hang in case of SQS issues.
        
    # Maximum number of seconds to retry high level SQS errors. right_aws
    # automatically retries using exponential back-off.
    aws_reiteration_time: 2
        
    # Low-level HTTP retry / timeout settings.
    http_retry_count: 2
    http_retry_delay: 1
    http_open_timeout: 2
    http_read_timeout: 10

Now start the Workling Client:

script/workling_client start

That’s it!

There are still some caveats, such as the fact that messages are currently deleted from the queue at the beginning of processing rather than at the end (unfortunately Workling currently doesn’t provide the necessary hooks). This is good enough for us (we’re not relying on our background jobs for anything highly critical), but you probably don’t want to build your financial transaction processing on top of this… If there’s demand for this, I may try to extend Workling at some point to fix this issue.

Please leave a comment if you find this useful or have any other feedback. Also let me know if you encounter any bugs, or better yet, update my test case to reproduce the issue or send me a Github pull request with your fix. :)