Last year at TaskRabbit, we decided to go headlong into this Service Oriented Architecture thing. We ended up with several Rails and other Ruby apps that loosely depended on each other to work. While this was mostly good from a separation of concerns and organizational perspective, we found effective automated testing to be quite difficult.

Specifically, we have one core platform application whose API is the most used. We also allow other apps to have read-only access to its database. Several of the apps are more or less a client to that server. So while the satellite apps have their own test suite, the integration tests can only be correct if they are exercising the core API.

To handle this use case, we created a gem called Offshore. A normal test suite has factories and/or fixture data and it uses database transactions to reset the data for every test. Offshore brings this the SOA world by providing the ability to use another application’s factories from your test suite as well as handling the rollback between tests. Through the actual exercising of the other application as well as a simplified usage pattern, we found that we trusted our integration tests much more that alternative approaches.

System

What we have are several apps based on use case, all using a core platform. There are web apps for each of the participants in our marketplace: workers, consumers, and businesses. We also have iPhone and Android apps along those same lines.

The apps communicate in a few ways. All of them use the platform’s API synchronously. The web apps use Resque Bus to subscribe and publish asynchronously. We also allow the web apps to read the platform database, but not write.

Use Case

The APIs and data involved are basically about tasks and users in the system and their various state transitions. For example, our business app signs up a user and post a tasks using the API. The user will then be on the their dashboard seeing a list of tasks in various states, conversations with the workers, payment details, etc.

This dashboard has many, many (many) possible combinations of what it can show after factoring in all the types of tasks and their states. To have an effective test suite for this one page, we’ll need several tests that put it in those various states. The page is mostly Javascript driven.

The other thing we want to test are flows that got the user into those states. For example, can the user post a task, receive some bids, chat back and forth, select a worker, have the worker mark it complete, pay for it, and rate the worker without any problems?

The first thing that we did was what we would have to do if this was a true “external” service. We stubbed the responses. We made our Javascript tests in Konacha and our Ruby tests used rspec and vcr. Both used “recordings” of API responses to “stub” the platform. This had a few problems.

First, there was still the question of what server to use. Should it be the local one or staging one or something else? Second, you kind of had to get it right the first time if there was any state involved in the test. If test signed up a user and then posted a task, for example, and it failed just a bit the first time, then it would fail harder the second time because that database was not reset (and the user existed when it went to sign up). Third, I was concerned that this easily get out of date with our evolving platform. Finally, if anything was read directly from the database, this would not work at all (as the data wouldn’t actually be there).

Offshore Pattern

When using Offshore, we actually run the platform server and the app uses that instead of stubbing. Both have the offshore gem installed.

Before a test, the app tells the platform that it’s about to start a test. It is then given a token that lets it make other requests to it during the test. It will create objects using factories, make regular API calls, and read/write to the platform database as required. It can do it’s own checks throughout to see if the test should pass. When it’s done, it let’s the platform know.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

describe "Integration Test", type: :request do
  it "should work as expected" do
      Offshore.test.start(example)

      user   = users(:billy)
      worker = users(:robby)

      task   = FactoryOffshore.create(:task_posted, :user_id => user.id)
      offer  = FactoryOffshore.create(:offer, :worker_id => worker.id)

      task.state.should == "opened"

      visit "/tasks"
      click_on "Hire!"

      task.reload.state.should == "assigned"
      task.worker_id.should == worker.id

      Offshore.test.stop
  end
end

The point is that this is more or less the same test that you’d write if you were using factories and capybara in a regular test suite.

Server

The server app is the one with the factories and the database that your app needs to work. For Rails, add this to your Gemfile:

1
2
3

group :test do
  gem 'offshore'
end

We also add the rake tasks to your Rakefile

1
2
3
4
5

begin
  require 'offshore/tasks' if defined?(Offshore)
rescue

end

You might need something like this to your test.rb application config:

1
2

Offshore.redis = "localhost:6379"
Offshore.enable! if ENV['OFFSHORE'] == 'true'

Then run something like this on the command line

$ rake offshore:startup
$ OFFSHORE=true rails s thin -e test -p 6001

In you want it anything but a blank database, you must create a rake task called offshore:seed that creates the test database referenced in the database.yml file in the test environment. Something like this would work:

1
2
3
4
5
6
7
8
9
10
11
12
13

namespace :offshore do
  task :preload do
    ENV['RAILS_ENV'] = "test"
  end
  task :setup => :environment

  desc "seeds the db for offshore gem"
  task :seed do
    Rake::Task['db:migrate'].invoke
    Rake::Task['db:test:prepare'].invoke
    Rake::Task['db:seed'].invoke
  end
end

The :preload and :setup steps will be invoked in that order before your :seed call. They are actually unnecessary here, but shown in case you have something more complex to do.

Client

The client app is the one running the tests.

The same thing in the Gemfile:

1
2
3

group :test do
  gem 'offshore'
end

The Rspec config looks likes this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

config.before(:suite) do
  Offshore.suite.start(:host => "localhost", :port => 4111)
end

config.before(:each) do
  Offshore.test.start(example)
end

config.after(:each) do
  Offshore.test.stop
end

config.after(:suite) do
  Offshore.suite.stop
end

You could also do this based on tags if you didn’t need this behavior in all your tests.

How It Works

The rake offshore:startup calls your seed rake worker and then takes a snapshot of the database. We use fixture data and by making a “template” of the database, Offshore is able to copy it into place when needed to create the illusion of “transactional” behavior.

So now we run the server and have it enabled. Requiring the gem required a Railtie that added the Offshore::Middleware which will respond to the requests that it serves (factory_create, suite_start, suite_stop, test_start, test_stop). Calling Offshore.enable! will tell it to handle the requests. Note you can just add Offshore::Middleware if you want to use this with Sinatra or other Rack apps.

When it receives the suite_start command, it sets everything up to run and records who is running. The main things to set up are the database and Redis lock. Offshore uses Redis to make sure only one test is using it’s database at a time, further simulating the notion of transactions.

The test_start command acquires the lock and copies the template to be the real database. If the lock is not acquired, the server will return an error code to the client to say wait. The client will poll until it’s available.

The client can now call factory_create or real APIs as much as it wants. The changes are made in the real database.

The test_stop command releases the lock.

We can go through many tests, calling start_test each time to reset the database so we get a fresh copy.

At the end, the suite_stop method notes that this client is no longer running.

Deployment

Developers can run this locally fairly easily using the instructions above. We have also deployed this to a server that auto-refreshes based on our master branch. This allows our continuous integration service called Tddium to use Offshore as well. Multiple branches can be building at the same time and it works out because of the locks.

Summary

We’ve gained a lot more confidence in the overall performance of our environment by exercising both the server and client app in parts of our test suites. Offshore makes this possible by enabling factories and database “transactions” across apps and threads. There’s plenty more things to make this even better to improve performance, but we thought it was an interesting pattern. Let us know if you find it helpful.

Note that while this is the best approach we could come up with for multiple apps, in our newer project, we chose to have a single app with multiple engines. This was in part to make tricks like that Offshore does unnecessary.