At TaskRabbit, we have gone through a few iterations on how we make our app(s). In the beginning, there was the monolithic Rails app in the standard way with 100+ models and their many corresponding controllers and views. Then we moved to several apps with their own logic and often using the big one via API. Our newest project is a single “app” made up of several Rails engines. We have found that this strikes a great balance between the (initial) straightforwardness of the single Rails app and the modularity of the more service-oriented architecture.

We’ve talked about this approach with a few people and they often ask very specific questions about the tactics used to make this happen, so let’s go through it here and via a sample application.

Rails Engines

Rails Engines is basically a whole Rails app that lives in the container of another one. Put another way, as the docs note: an app itself is basically just an engine at the root level. Over the years, we’ve seen sen engines as parts of gems such as devise or rails_admin. These example show the power of engines by providing a large set of relatively self-contained functionality “mounted” into an app.

At some point, there was a talk that suggested the approach of putting my our functionality into engines and that the Rails team seemed to be devoting more and more time to make them a first class citizen. Our friends at Pivotal Labs were talking about it a lot, too. Sometimes good and sometimes not so good.

Versus Many Apps

We’d seen an app balloon and get out of control before, leading us to try and find better ways of modularization. It was fun and somewhat liberating to say “Make a new app!” when there was a new problem domain to tackle. We also used it as a way to handle our growing organization. We could ask Team A to work on App A and know that they could run faster by understanding the scope was limited to that. As a side-note and in retrospect, we probably let organizational factors affect architecture way more than appropriate.

Lots of things were great about this scenario. The teams had freedom to explore new approaches and we learned a lot. App B could upgrade Rack (or whatever) because it depend on the crazy thing that App A depended on. App C had the terrible native code-dependent gem and we only had to put that on the App C servers. Memory usage was kept lower, allowing us to run more background workers and unicorn threads.

At TaskRabbit, we are using Resque to do our background job processing. We’ve also gone one step further and used Redis and Resque to create an asynchronous message bus system that we call Resque Bus.

Redis / Resque

Redis is a single-threaded in-memory key/value store similar to memcached. Redis has other features like pub/sub and more advanced data structures, but the key feature that makes it an ideal storage engine for a queue and a message bus is that is can perform atomic operations. Atomic operations are the kind of operations you can expect to do on in-process data (like Array.pop or Array.splice) but in way that keeps the data sane for everyone connected to the database.

Resque is a background queue built on top of Redis. There seems to be other options out there these days, but we are pretty happy with Resque and associated tools/ecosystem. There is plenty of code in the resque codebase, but it all comes down to inserting json the queue, popping, and executing code with that as an input.

Resque Bus

Resque Bus uses Resque to create an asynchronous message bus system. As we have created more applications with interdependencies, we have found it helpful to create something like this to loosely couple the worlds. There are several other possible solutions to this problem, but I really felt that it was important to use something that our team understood well for this piece of infrastructure that we could easily modify and maintain.

It’s not exactly a best practice, but sometimes it comes up and I couldn’t find any material on the Internet about how to make it work. The situation is that you’re saving your record and it’s after_save and now you’ve decided that you don’t want to save it at all. What to do? The short answer is to raise ActiveRecord::RecordInvalid.new(self) and run away.

How did you get here?

This seems most relevant to syncing scenarios. For example, when a Task is created, we need to create a parallel record in an external system. When it is necessary, it has to happen and the Task should not exist (or be saved) without it. I could make that call in a before_save callback and add a validation error if it didn’t work. However, if the rest of the validations don’t work, then there is no taking back that call. Everything else in a SQL transaction, but not other systems. I had some success with making absolute sure that it was the last before_save and that worked out for a while. Then we needed to send the id of the Task to the external system. This just does not exist before the save actually occurs the first time. So I wanted to put it in an after_save callback.

What to do?

The thing to note in this case, though, is that after_save is still in the SQL transaction. So if we freak out enough, it will roll the whole thing back. The trick is freaking out in the right way.

Returning false no longer seems to stop things. I swear that used to happen in older (< 3) versions of Rails. Raising most errors will stop the transaction but also crash the system. The first one that I tried was ActiveRecord::Rollback and it worked just fine in that it did not save and did not crash, but this test that I had was failing.

1

task.save.should == false

Now, I wouldn’t have even have caught this if I did what I usually do which would be to use the task.save.should be_false rspec helper. This is because raising ActiveRecord::Rollback ended up in the save call returning nil. That would usually be fine, but I wanted to get it just like normal.

If you take a look at the ActiveRecord code for save, the answer reveals itself.

1
2
3
4
5

def save(*)
  create_or_update
rescue ActiveRecord::RecordInvalid
  false
end

By raising ActiveRecord::RecordInvalid we treat it like a validation error and it has the expected behavior. I went ahead and added an actual error to seem even more like the normal case. Final code:

1
2
3
4
5
6
7
8
9
10
11
12

class Task < ActiveRecord::Base
  after_save :sync_with_external

  def sync_with_external
    response = External.sync!(id: self.id, info: self.info)
    if response.error?
      self.errors.add(:base, "There was a problem, etc ...")
      raise ActiveRecord::RecordInvalid.new(self)
    end
    true # I still do this out of superstition
  end
end

Caveats

This main issue that comes up is that you only get to have one of these to be absolutely sure everything is fine. If there were two of these external services, you’d end up with the same original problem. I guess, you should put and your flakiest ones first or try to get out of it altogether.

Also note that or non-immediately-critical syncing (like search indexing), the right spot for these types of things are in after_commit where I would queue up a background job with retry logic. That would be outside of the SQL transaction and actually be needed to prevent timing issues in that background thread.

We are making lots of new apps at TaskRabbit and have many shared components. All of this stuff (gems, best practices, test setup, etc) needs to get into the new project quickly so the team can focus on the business of providing the intended functionality. My first attempt to solve this problem was to create a Rails app template with all that stuff, but switched to a new way that involves a true template application.

Conventional app templates seem to work for a lot of people and Rails Composer looks to be a good option in this space. This method works by running Ruby code that puts down files. This level of indirection is likely a positive thing when projects vary (rspec one time, test-unit the next). However, we’re trying as hard as we can to be consistent across apps, so when I found that this indirection made working with the templates more difficult, I decided that it was not worth it.

The alternative method is pretty simple. I made a Rails app called Warren) that was exactly what I wanted to see new projects use. It contained all our our gems for authentication, shared styles, inter-app communication, and developer tools. You could basically log in, log out, and go to a page where you saw who was logged in. It had its own test suite to test these things along with all the normal stuff: yml files, initializers, .rvmrc, .gitigore, ApplicationController, favicon, etc. It was simply a working Rails app.

The only addition is the app.rake file that I put in the /lib/tasks directory. That provides this functionality:

% rake app:create[new_name]

This will create a new app called NewName in a peer folder to Warren. The full process looks something like this:

localhost:warren brian$ rake app:create[new_name]

localhost:warren brian$ cd ../new_name
=== RVM Message ===
Do you wish to trust this .rvmrc file? (/Users/brian/taskrabbit/new_name/.rvmrc)
y[es], n[o], v[iew], c[ancel]> yes
Using /Users/brian/.rvm/gems/ruby-1.9.3-p194 with gemset new_name

localhost:new_name brian$ bundle install
Fetching gem metadata from http://rubygems.org/......
Installing all-the-things (3.9.2) 
Your bundle is complete!

localhost:new_name brian$ rake db:create db:migrate db:test:prepare

localhost:new_name brian$ rails s
>> Listening on 0.0.0.0:3000, CTRL+C to stop

And http://localhost:3000 works.

In Ruby, we often like very simple APIs, but this often comes at the price of thread safety and code flexibility. I’ve found that if you use a few tricks from the start, you can get the best of it all.

I recently did a project where I tried to use the VCR gem, but it went awry when working in multiple threads. This is a great gem that, like many of my own, falls into the trap of module/class level singleton configuration/execution.

This is approach is characterized by things like extend self in the top-level module and then having instance variables at that level. This is not to call out VCR specifically. it’s just my most recent example of hundreds of gems that take this overall approach.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

module VCR
  extend self

  def current_cassette
    cassetes.last
  end

  def configure
    yield configuration
  end

  def configuration
    @configuration ||= Configuration.new
  end

  private

  def cassettes
    @cassettes ||= []
  end
end

When operating on multiple threads, things get wacky because of this because they are sharing this current_cassette and writing to the associated file. You end up with recordings on top of each other.

I am inclined (and some say over-inclined) to use singletons to do something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

class VCR::Client
  def current_cassette
    cassetes.last
  end

  def configure
    yield configuration
  end

  def configuration
    @configuration ||= Configuration.new
  end

  private

  def cassettes
    @cassettes ||= []
  end
end

module VCR
  extend self

  delegate :current_cassette, :configure, :configuration, :to => :default_client

  def default_client
    @default_client ||= Client.new
  end
end

I got pretty sidetracked during my 2012 Review and ended up making a new tool called Hubtime. It lets you graph your activity in a variety aways across Github repositories.

The inspiration came from Github’s own reports, but I made Hubtime because those reports are only at a repository level. I was looking to see patterns across the many repositories that I worked on over the year. It seems that Github agrees to some degree because they have since launched metric overviews on user profile pages.

A 42 day streak! Hubtime has the data to make Hubtime do the same graph, but right now focuses more on reproducing the Github experience that is only now per repository.

$ ./hubtime.rb graph commits --months 3

Graphing commits, additions, deletions, or impact can also be done showing the magnitude by repository.

$ ./hubtime.rb graph commits --months 3 --stacked

This was the first purely command line Ruby project that I have done. In addition to that, I learned several new things and tried some new approaches during development.

We didn’t go anywhere over the holidays and had a nice Christmas at home. It’s possible that I’m incapable of taking a vacation at all, but this is particularly true when on a stay-cation. Other than Christmas day itself, I was coding on a normal schedule for most of the week.

My daughter (4.5) was around most of the week too, often with the normal babysitter that comes for my younger son. So all week, our house had me coding, my wife editing, and her hanging out with the babysitter. She even brought us lunch like it was restaurant including a fork wrapped up in a napkin.

On January 1st, I also took the day completely off and asked her what she wanted to play. “Not now, I’m working,” she said. She was “playing work,” which involved writing out a dense set of letters on white sheets of paper.

“There is a lot to do and I have to work, even on family days.”

And she went back to scribbling like crazy. This all says something about me as a person and a parent. I’m not sure exactly what, but it’s probably not good.

It hurts because I actually think I’m doing a pretty good job. I leave early (by engineer-at-startup standards) to have dinner and hang out with her and only work again after she is asleep. I’m not sure I could have more “balance” anyway. To me, the term suggests a healthy ratio between what you are forced to do because of work and the things of “normal life.” I might be intrinsically unbalanced, but I feel lucky that my normal life has a lot of things that I love doing which centers mostly around building things for people. For better or worse, I’m thinking about this stuff all the time, anyway.

It likely comes with my being so visibly inaccessible all week. Regardless, I didn’t have any profound realizations at that moment or anything. Instead, I decided to plant a behavioral seed for later.

“I understand. I’ll let you get to it. Let me know when you can play.”

I honestly can’t remember anything from January or even October very specifically. It sure felt like I was busy as we doubled the TaskRabbit engineering team and scaled the site to get a lot more stuff done for real people in the real world.

I can’t even decide what the best way is to see what I did in January. I could look at my email or calendar, but I strive not to make those the center of my world. Maybe in Evernote. My git commits are probably a bit too granular, but maybe there is something in the aggregate.

Most striking is the almost non-existence of code between July and October. We were growing the team during this time and I made the conscious choice to try on more of an “engineering manager” role as opposed to a “tech lead” kind of role. Judging by the large spike in October, it’s clear the experiment got cut short. Around this time, I made the biggest change of the year: transitioning from VP of Engineering to Chief Architect.