AMQP: Integrating Spring and Rails with RabbitMQ

Why?

Spring web applications tend to grow and become bigger and more complicated over the time. So it's often a good choice to source out a specific responsibility and process it in an other app made for the specific job. You can build a clean interface to the new app and maybe have an extra team just working on it. Plus you have a free choice of the programming language. I like Ruby.

Another big reason is that you don't want to do all computations in the time of one request of your Spring webapp. If something could be processed later, why not improve the response time and latency of your web app by asynchronously handing the job over to another app.

Maybe you guessed it: This can easily done with the AMQP protocol and one or many RabbitMQ broker. AMQP stands for Advanced Message Queuing Protocol which is an open standard and RabbitMQ is an excellent open sourced server or broker that implements this protocol. With AMQP you can asynchronously send messages in a standardized protocol that is supported by the vast amount of programming languages. There even is a spring project called spring-amqp and of course a ruby gem called amqp by tmm1. To get an idea how RabbitMQ works I encourage you to check the projects homepage.

Getting started

First install RabbitMQ. Be sure to install RabbitMQ of version greater than 2. Otherwise you get "Protocol Mismatch" Errors.

If your on a Mac and have brew installed its easy like this:

$ brew install rabbitmq

Next install the amqp gem and start rabbitmq. You can use the following scripts to verify that sending and receiving messages over AMQP works:

# consumer.rb
require "rubygems"
require "mq"

AMQP.start do
  queue = MQ.queue('hello.world.queue')
  
  queue.subscribe do |word|
    puts word
  end
  
end

# producer.rb
require "rubygems"
require "mq"

AMQP.start do
  queue = MQ.queue('hello.world.queue')
  
  i = 0
  EM::add_periodic_timer(1) do
    queue.publish "hello world #{i+=1}"
  end
  
end

The Consumer simply subscribes to the hello.world.queue. If a message arrives the subscribe block gets called and prints the message to the console. The Producer just pushes a message every second to the queue. Since AMQP already uses EventMachine, you can use all features of EventMachine like add_periodic_timer and so on.

If everything works fine, you should see the Consumer receiving "hello world"-messages.

Now we build a new Rails 3 app and configure it to process messages.

$ rails new consumer_app

You need to put the dependencies for tmm1's amqp gem and the thin server in your gemfile:

gem 'amqp', :require => 'mq'
gem 'thin'

Why do we use thin as webserver? Simply because thin provides a running EventMachine reactor. Without this reactor EventMachine will block the current thread and no HTTP request would be processed. If you like to use an other server than thin you need to activate the EventMachine reactor manually. To do this, just put

Thread.new { EM.run }

in an initializer.

Now, create a Model called example_message.rb with following content:

# app/models/example_message.rb
class ExampleMessage

  def self.start_listen
    AMQP.start do  
      MQ.queue('hello.world.queue').subscribe do |msg|
        puts msg
      end
    end
  end

end

The listen-method can be triggered by a Rails initializer. But the AMQP block will block during the boot process of Rails when we call ExampleMessage.listen directly. We can get around this by using EventMachine to call it later in the next available time slot.

# config/initializers/example_message.rb
EM::next_tick do
  ExampleMessage.listen
end

Notice: My colleague Tobias Sunderdiek found out, that this only works for Ruby Version greater than 1.8.7 with a patchlevel around 320. It will not work with the current Ruby Enterprise Edition, which has a patchlevel of 253.

Now, if you start your Rails app and the Producer script you should see your app receiving messages.

Lets try to switch from the simple Producer script in Ruby to a simple Producer built in Java that uses the same queue. Luckily the Spring-AMQP project comes with an example that exactly does this called helloworld. You can fetch the Spring-AMQP sources using git as shown here:

$ git clone https://github.com/SpringSource/spring-amqp.git

To compile and run the helloworld example navigate to samples/helloworld and type:

$ mvn compile
$ mvn exec:java -Dexec.mainClass="\
  org.springframework.amqp.helloworld.async.Producer"

Those Spring-AMQP examples are a great starting point to find out how Spring-AMQP works. From there it is a easy step to build a Service that does the message sending for a specific use case. Here is a simple Service that uses the RabbitTemplate to send messages:

@Service
public class HelloWorldMQService {

    @Autowired
    private RabbitTemplate rabbitTemplate;

    public void sendMessage(int i) {
        rabbitTemplate.convertAndSend("hello world "+i);
    }
}

The RabbitTemplate must be configured before it can be used. To do this you need a AbstractRabbitConfiguration as shown below:

@Configuration
public class HelloWorldConfiguration 
        extends AbstractRabbitConfiguration {

    protected final String helloWorldQueueName = "hello.world.queue";

    @Override
    public RabbitTemplate rabbitTemplate() {
        RabbitTemplate template = 
            new RabbitTemplate(connectionFactory());
        template.setRoutingKey(this.helloWorldQueueName);
  
        return template;
    }

    @Bean
    public ConnectionFactory connectionFactory() {
        SingleConnectionFactory connectionFactory = 
            new SingleConnectionFactory("localhost");
        connectionFactory.setUsername("guest");
        connectionFactory.setPassword("guest");
        return connectionFactory;
    }

Now you can integrate this AMQP service with your Spring webapp.

If you like to dig deeper in AMQP, Ruby and Spring I encourage you to check out this article and this presentation on infoq.com.

Cheers,
Arbo

Testing SOAP Webservices with RSpec

SOAP webservices are widely used in enterprise environments. Although they feel a bit clumsy in comparison to slim REST services, sometimes you have to deal with them.

The great thing is, to test such a service you are often free to use any tool you like. I like RSpec!

To query a web service you just need a few lines of code. I recommend Savon as SOAP client. It is used as shown here:

require 'rubygems'
require 'savon'

WSDL_URL  = 'http://www.webservicex.net/geoipservice.asmx?wsdl'

client = Savon::Client.new WSDL_URL
response = client.get_geo_ip do |soap|
  soap.body = { "wsdl:IPAddress" => "209.85.149.106" }
end
puts response

The response object can be converted to hash with the to_hash method, so you can fetch all values simply like you would do it with any other hash.

Now, the rest should be easy and is just a normal RSpec test:

require 'rubygems'
require 'savon'

WSDL_URL  = 'http://www.webservicex.net/geoipservice.asmx?wsdl'

RETURN_CODE_OK    = "1"
RETURN_CODE_ERROR = "0"

describe "Geo IP Webservice at #{WSDL_URL}" do
  
  # helper method
  def get_geo_ip_result ip
    response = @client.get_geo_ip do |soap|
      soap.body = {"wsdl:IPAddress" => ip}
    end
    response.to_hash[:get_geo_ip_response][:get_geo_ip_result]
  end
  
  before :all do
    @client = Savon::Client.new WSDL_URL
  end

  it "should yield a country name" do
    result = get_geo_ip_result "209.85.149.106"
    result[:country_name].should_not be_nil
    result[:return_code].should eql(RETURN_CODE_OK)
  end
 
  it "should return error for malformed ip address" do
    result = get_geo_ip_result "not.an.ip.address"
    result[:return_code].should eql(RETURN_CODE_ERROR)
  end
 
  it "should fail if no ip address is submitted" do
    lambda { @client.get_geo_ip }.should raise_error
  end

  # ...

end

Happy testing!

EDIT:

@dbloete pointed me to the fact that with RSpec 2 you can expect errors even more readable:

it "should fail if no ip address is submitted" do
  expect { @client.get_geo_ip }.to raise_error
end

CouchDB: Using List Functions to sort Map/Reduce-Results by Value

I just found out that it is possible to sort the result of Map/Reduce with a list function.

Let's take the simple example that you want to count all documents grouped by a field called type. The following map function emits the values of the type fields of all documents:

function(doc) {
  emit(doc.type, 1);
}

To sum up the documents with the same value in the type field, we just need this well-known reduce function:

function(key, values) {
  return sum(values)
}

By default CouchDB yields the result ordered by the keys. But if you want to order the result by occurrences of the type of the document you either have to sort it in your app or you use a list function like this:

function(head, req) { 
  var row
  var rows=[]
  while(row = getRow()) { 
    rows.push(row)  
  } 
  rows.sort(function(a,b) { 
    return b.value-a.value
  }) 
  send(JSON.stringify({"rows" : rows}))
}

If you save the list function as sort and the Map/Reduce-functions as count together in a design document, you can fetch your sorted result like this:

curl http://.../design-doc/_list/sort/count?group=true

Of course there are other options to sort a view result. I didn't found much documentation on this topic, but this thread at stackoverflow is very informative.

Back to the couch - Cheers!

jQuery meets CoffeeScript

Disclaimer: This article is pretty old.

I'm just amazed how brilliant jQuery and CoffeeScript are working together. JQuery promises "write less, do more", but with the clean syntax of CoffeeScript you can be even more concise!

Here is a quick example:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta http-equiv="content-type" content="text/html; charset=utf-8" />
  <title>jQuery meets CoffeeScript</title>
    <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>
    <script src="http://jashkenas.github.com/coffee-script/extras/coffee-script.js"></script>
  
    <script type="text/coffeescript">

      show_message = (msg) -> 
        $('#message').hide().text(msg).fadeIn(2222, 
          -> $('#message').append('!') 
        )
              
      $ -> show_message "world"
      $('#message').click -> show_message "you"
      
    </script>

</head>
<body>
  <h1>hello <span id="message"></span></h1>
</body>
</html>

Here you see it running. Just click "world" to fire the click event.

hello

Just have a look to the JavaScript version:

var show_message = function(msg) {
  return $('#message').hide().text(msg).fadeIn(2222, function() {
    $('#message').append('!');
  });
};
$(function() {
  show_message("world");
});
$('#message').click(function() {
  show_message("you");
});

Happy coding!

Getting started with node.js and CoffeeScript

After a couple of month the voices saying Node.js is awesome getting louder and louder, I decided to give it a try.

There is something different I never played with but I absolutly need to: CoffeeScript. I was quite amazed to find out, that those two new shiny things fit and work perfectly with one another. If you like to get Node and CoffeeScript up and running just follow my little tutorial. Its written for Mac but it should work for Linux too.

To get started first install Node.js. So lets grep the latest version directly from github at http://github.com/ry/node or download the tar-archive from http://nodejs.org/#download. To build and install Node type:

./configure
make
sudo make 

To install CoffeeScript go and clone the Node Package Manager from http://github.com/isaacs/npm.

git clone http://github.com/isaacs/npm.git

switch to the nmp directory and type

sudo make

The latest stable CoffeeScript-version will be installed to you computer. Now you have binary called "coffee", which just starts your CoffeeScripts with Node.

If you're a TextMate fangirl or fanboy you may want to install the CoffeeScript TextMate bundle. It can be found at http://github.com/jashkenas/coffee-script-tmbundle.

Ok, now you ready to play. Here is a hello world server written in CoffeeScript:

server = require('http').createServer (request, response) ->
    response.writeHead 200, {'Content-Type': 'text/plain'}
    response.end 'Hello World'
server.listen 8124
console.log 'Server running at http://0.0.0.0:8124/'

Save it as hello_world.coffee and run it with:

coffee hello_world.coffee

Open http://0.0.0.0:8124/ to see "Hello World" in your Browser.

Just take a look to the JavaScript version:

var server = require('http').createServer(function (request, response) {
  response.writeHead(200, {'Content-Type': 'text/plain'});
  response.end('Hello World');
})
server.listen(8124);
console.log('Server running at http://0.0.0.0:8124/');

Although it's a short and dead simple script, for me the Coffee version is much cleaner and readable.

Cheers!