Amping RSpec

Or rather, RSpecing Amp.

One of the challenges of Amp is the variety of repository format plugins, all of which are expected to obey a common API. It’s especially challenging since there is little test coverage. My solution to this is to create an Amp repo spec in emulation of the Ruby spec project which specifies the behavior of a conforming ruby implementation.

Shared Tests

Creating a common test suite for a number of as yet unknown classes and objects presents it’s particular challenges. RSpec has a couple of features which I’m using to approach the problem.

First and foremost, it’s possible to define “shared example groups” that can be included in another test. Though I have no reference points, I suspect I’m doing this on an especially large scale. Each repository format is represented by suite of classes, each with a large number of API methods. My current strategy is to have a shared example group for each class, which uses directory listing and an assumed file-name/test-name correspondence to load up second set of shared example groups, one for each method in the class API.

require 'amp_repo_spec/amp_spec_helper'

shared_examples_for 'local repository' do
  include AmpRepoSpec::Helper

  subject {described_class.new(tempdir)}

  Dir.glob(File.join(File.dirname(__FILE__), 'local_repository/*.rb')).each do |file|
    require file
    it_should_behave_like 'local_repository#' + File.basename(file, '.rb')
  end
end

What’s your name again?

Another challenge is that, when writing a normal test, you know the name of the thing you are testing. RSpec allows an implicit subject, so that having said describe Foo, you can then assume you are testing an instance of Foo. However, repositories, and many other classes, have to be initialized with a parameter, so it’s still necessary to use the subject method. Doing this in a completely generic manner requires using the described_class method to get the original implicit subject to instantiate.

Unfortunately, this only takes you so far. Repositories are represented by systems of classes. So, for instance, to create a staging area, one has to pass an initialized repository. Relying only on the implicit subject from the hosting test won’t tell you about related classes. My solution so far is declare ModuleUnderTest as an alias to a namespace under which standard names (LocalRepository, StagingArea, and so forth can be found) Another idea I just had would be to have each class declare it’s related classes, so that e.g. Amp::Core::Repositories::Rugged::StagingArea::LocalRepository (that is, described_class::LocalRepository) would point to Amp::Core::Repositories::Rugged::LocalRepository Doing it as methods (local_repository_class) would make it easier to stub out on a case-by-case basis.

Implicated

Despite a few troubles with names, I have been playing around with the implicit subject style of testing. I used to write tests like this:

it "should bounce" do
  ball = Ball.new
  ball.bounce?.should == true
end

Often times I would start seeing common set up code and pull that into a before block. Over time I started using describe blocks to apply different setup to different sets of tests. Lately I’ve been using subject and/or the implicit subject to cut out extra verbiage.

describe "when thrown" do
  subject {Ball.new}
  its(:bounce?) {should be}
end

Custom Matchers

Another piece of new testing technology is custom matchers. I found that some tests required multiple steps or long chains of methods to get to some testable attribute. I’d heard of RSpec’s custom matchers before, but always steered clear on assumed complexity. There is a small amount of ceremony to it – and quite a bit more if you get into custom error messages and all other optional methods.

RSpec::Matchers.define :contain_the_file do |file|
  match do |actual|
    File.exist?(File.join(actual, file))
  end
end

it {'/tmp'.should_not contain_the_file('passwords')}
its(:root) {should contain_the_file('Ampfile')}

Temporary Insanity

One of the special challenges of dealing with a source control system is that it involves the filesystem. At one point I thought out nice it would if all file systems calls ran through an object we could mock out during testing. However, one particular item that has my attention right now is Rugged e.g. libgit2, a C reimplementation of Git, focused on embedding in other applications. It’s not Ruby, and it’s going to read or write the filesystem whether I want it to or not.

This leaves me with the problem of ensuring a clean temporary directory for each test. Some of tests I inherited used the project devver-construct to set up a directory for each and every test. That was appropriate for the tests at hand, but I wanted to use contexts with a setup and multiple tests on each repository without having to rebuild the repo for each inquiry.

I managed to figure out an arrangement of before and after triggers using construct to actually generate the temporary path, though it takes some care to figure out where to define the methods and how to store the path so it can be used in tests and cleaned up afterwards. When I started on the second class to test, I extracted this to a shared module. RSpec is fairly well set up to look clean in the typical case, but once you start trying to make an add-on module, you start to realize just how many gymnastics it’s going through to mix class context and instance context so fluidly. For instance:

  • In order to set up a before/after on module inclusion, I had to call before and after on the argument to included
  • To set up helper methods, I had to create a ClassMethods module and extend it.
  • Then in those class methods, I can use before and company without an explicit receiver again.
  • Methods defined in the main module body are for the test instances (inside it)

All this makes me wonder if I should be using extend and an InstanceMethods submodule, but include is more conventional in Ruby.

The repository factory

The latest innovation in the Amp repo spec doesn’t have much to do with the test framework at all. After finally getting the core setup operations (init, add, commit) working, I was able to start testing everything else. A while at this and the repetitious patterns started to emerge, such as needing to steps to add a file (create the file, then add it to the repo) I abstracted this out into a ‘repo factory’ that provided terse DSL to cover over some of the details (such as the distinction between current-working-directory and repository-directory)

in_a_new_repo do
  add 'file.one'
  add 'file.two'
  commit
  modify 'file.one'
  modify 'file.two'
end

Stubbed my toe

RSpec includes a mocking/stubbing facility by the name of double. However, it doesn’t work quite how I wanted. I had the impression that ‘mocking’ was setting expectations on an object to make sure interactions were happening as expected, in contrast to ‘stubbing’ where you just trying to e.g. not access the database, and don’t really care what the subject does with it. Unfortunately, RSpec’s doubles keep count of how often stub methods are called and throw a fit if it’s more than once. I had to revert to PORO (plain old Ruby objects) to stub out tangential functionality.

Documentation?

All through this, I was trying to learn a lot of new features of RSpec, but the best source I could find was relish, an online version of the Cucumber specs for RSpec itself. While the tests are (hopefully) accurate, they don’t necessarily provide the best teaching examples. I assume I haven’t just missed something here, since David Chelimsky acknowledged the documentation could use some help (as in ‘please help’) when he talked about RSpec briefly.

Isn’t there a book about that?

Come to think of it, I’m probably getting deep enough into RSpec that I should probably look into the recently published RSpec book, which will hopefully have some more illustrative examples.

Posted Sunday, February 6th, 2011 under Devlog.

Tags: ,

Comments are closed.