Antilles: Testing CLIs that use Web Services

We’re happy to announce our first OSS release, the antilles gem!

Antilles is a companion to aruba that makes it easier to test CLIs that use web services by spawning a local stub HTTP server.  It’s an alternative to fakeweb, for cases like aruba, where fakeweb can’t be used because the process being tested can’t have its network conveniently monkey-patched.

A major component of Tddium is the tddium command line interface.  When we started building the tddium command, we wanted to write high-level tests, and we saw great examples in the cucumber tests written for the rspec command.  These features use aruba to automate testing CLIs by setting up a sandbox directory and defining a number of convenient cucumber steps for running and interacting with commands.

We wanted a test like this:

@mimic
Feature: Login command

  Scenario: Interactively log in successfully
    Given the user can log in and gets API key "apikey"
    When I run `tddium login` interactively
    And I type "foo@example.com"
    And I type "barbarbar"
    And the console session ends
    Then the output should contain:
    """
    Logged in successfully
    """
    And the exit status should be 0
    And dotfiles should be updated

But we quickly realized we couldn’t do this with aruba alone.  Our CLI relied heavily on our web service at tddium.com.  In order to test properly, we’d need to stub out the web service.  We’d started using fakeweb for other tests.  But when we tried to make fakeweb work with cucumber/aruba tests, we found that the way aruba runs the command-under-test as a separate process, any network stubbing done in the test itself wouldn’t make any difference.

We had other things to worry about, like launching our service, so we chose what we knew – rspec – and postponed our thoughts of using aruba.   Maybe our rspec-fu was lacking, but the rspec tests we produced were massive, complex, and convoluted.  Our CLI uses Thor; each subcommand is an instance method.  There’s 150+ lines of shared stub, fakeweb, and exit status detection superstructure.  One 25 line method had 70 lines of rspec to cover all of its cases, on top of the 150 shared lines.  These tests were complete, but hard to follow and even harder to extend.

In an ideal world, we’d have the command-under-test to talk to a “fakeweb server” running in a thread controlled by the cucumber/aruba process.  Enter mimic, a stub HTTP server that can be forked and have its stub responses configured via an HTTP API.  Antilles forks a sub process running a mimic server, and provides a simple API to hook it into Cucumber tests.  Once the tddium command is configured to connect to the Antilles server instead of our internet-facing API server, voila, we have an end-to-end test that’s expressive and easy to maintain.



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 369 other followers