AMQP: Integrating Spring and Rails with RabbitMQ

Why?

Spring web applications tend to grow and become bigger and more complicated over the time. So it's often a good choice to source out a specific responsibility and process it in an other app made for the specific job. You can build a clean interface to the new app and maybe have an extra team just working on it. Plus you have a free choice of the programming language. I like Ruby.

Another big reason is that you don't want to do all computations in the time of one request of your Spring webapp. If something could be processed later, why not improve the response time and latency of your web app by asynchronously handing the job over to another app.

Maybe you guessed it: This can easily done with the AMQP protocol and one or many RabbitMQ broker. AMQP stands for Advanced Message Queuing Protocol which is an open standard and RabbitMQ is an excellent open sourced server or broker that implements this protocol. With AMQP you can asynchronously send messages in a standardized protocol that is supported by the vast amount of programming languages. There even is a spring project called spring-amqp and of course a ruby gem called amqp by tmm1. To get an idea how RabbitMQ works I encourage you to check the projects homepage.

Getting started

First install RabbitMQ. Be sure to install RabbitMQ of version greater than 2. Otherwise you get "Protocol Mismatch" Errors.

If your on a Mac and have brew installed its easy like this:

$ brew install rabbitmq

Next install the amqp gem and start rabbitmq. You can use the following scripts to verify that sending and receiving messages over AMQP works:

# consumer.rb
require "rubygems"
require "mq"

AMQP.start do
  queue = MQ.queue('hello.world.queue')
  
  queue.subscribe do |word|
    puts word
  end
  
end

# producer.rb
require "rubygems"
require "mq"

AMQP.start do
  queue = MQ.queue('hello.world.queue')
  
  i = 0
  EM::add_periodic_timer(1) do
    queue.publish "hello world #{i+=1}"
  end
  
end

The Consumer simply subscribes to the hello.world.queue. If a message arrives the subscribe block gets called and prints the message to the console. The Producer just pushes a message every second to the queue. Since AMQP already uses EventMachine, you can use all features of EventMachine like add_periodic_timer and so on.

If everything works fine, you should see the Consumer receiving "hello world"-messages.

Now we build a new Rails 3 app and configure it to process messages.

$ rails new consumer_app

You need to put the dependencies for tmm1's amqp gem and the thin server in your gemfile:

gem 'amqp', :require => 'mq'
gem 'thin'

Why do we use thin as webserver? Simply because thin provides a running EventMachine reactor. Without this reactor EventMachine will block the current thread and no HTTP request would be processed. If you like to use an other server than thin you need to activate the EventMachine reactor manually. To do this, just put

Thread.new { EM.run }

in an initializer.

Now, create a Model called example_message.rb with following content:

# app/models/example_message.rb
class ExampleMessage

  def self.start_listen
    AMQP.start do  
      MQ.queue('hello.world.queue').subscribe do |msg|
        puts msg
      end
    end
  end

end

The listen-method can be triggered by a Rails initializer. But the AMQP block will block during the boot process of Rails when we call ExampleMessage.listen directly. We can get around this by using EventMachine to call it later in the next available time slot.

# config/initializers/example_message.rb
EM::next_tick do
  ExampleMessage.listen
end

Notice: My colleague Tobias Sunderdiek found out, that this only works for Ruby Version greater than 1.8.7 with a patchlevel around 320. It will not work with the current Ruby Enterprise Edition, which has a patchlevel of 253.

Now, if you start your Rails app and the Producer script you should see your app receiving messages.

Lets try to switch from the simple Producer script in Ruby to a simple Producer built in Java that uses the same queue. Luckily the Spring-AMQP project comes with an example that exactly does this called helloworld. You can fetch the Spring-AMQP sources using git as shown here:

$ git clone https://github.com/SpringSource/spring-amqp.git

To compile and run the helloworld example navigate to samples/helloworld and type:

$ mvn compile
$ mvn exec:java -Dexec.mainClass="\
  org.springframework.amqp.helloworld.async.Producer"

Those Spring-AMQP examples are a great starting point to find out how Spring-AMQP works. From there it is a easy step to build a Service that does the message sending for a specific use case. Here is a simple Service that uses the RabbitTemplate to send messages:

@Service
public class HelloWorldMQService {

    @Autowired
    private RabbitTemplate rabbitTemplate;

    public void sendMessage(int i) {
        rabbitTemplate.convertAndSend("hello world "+i);
    }
}

The RabbitTemplate must be configured before it can be used. To do this you need a AbstractRabbitConfiguration as shown below:

@Configuration
public class HelloWorldConfiguration 
        extends AbstractRabbitConfiguration {

    protected final String helloWorldQueueName = "hello.world.queue";

    @Override
    public RabbitTemplate rabbitTemplate() {
        RabbitTemplate template = 
            new RabbitTemplate(connectionFactory());
        template.setRoutingKey(this.helloWorldQueueName);
  
        return template;
    }

    @Bean
    public ConnectionFactory connectionFactory() {
        SingleConnectionFactory connectionFactory = 
            new SingleConnectionFactory("localhost");
        connectionFactory.setUsername("guest");
        connectionFactory.setPassword("guest");
        return connectionFactory;
    }

Now you can integrate this AMQP service with your Spring webapp.

If you like to dig deeper in AMQP, Ruby and Spring I encourage you to check out this article and this presentation on infoq.com.

Cheers,
Arbo