Dr. Testlove or: How I Learned to Stop Worrying and Love Automated Testing* by Brent McNish
Posted: February 4, 2013 Filed under: Automated Testing, Continuous Deployment, Continuous Integration, Dev Ops, Performance, Ruby, Sauce Labs, Selenium, Software Engineering, Tddium Leave a comment »(A great post from one of our joint Sauce Labs / Solano Labs Customers at www.deliberator.com. Thanks Brent for sharing! Congrats guys on your Beta Launch!)
________
* with apologies to Stanley Kubrick
Pre-testoric
The image of the head-down coder hacking away Tasmanian Devil-like, paying – at best – lip service to writing tests is a thankfully less and less accurate cliché these days. But that wasn’t always the case. These guys (and girls) used to be the everywhere. They didn’t get into development to do testing! Look, the code works! Job done. Next feature, bring it on!
Yes, once upon a time the “who needs automated tests” dinosaurs ruled the earth. And I was one of them.
I began my web development career in a large IT consultancy. We had testers, and test managers, and elaborate test plans. They would catch the bugs. It wasn’t a developers job to test.
Then I left to co-found my first (bootstrapped) startup, as the sole developer, with a single co-founder. And in Bootstrapped-Startup-Land you don’t have testers, and test managers, and test plans. There is just you. And your code
And every line of code you write, someone, somewhere in another startup is also writing a line of code, and they might have the same idea as you. And they’re going to launch their feature before yours. And they’re going to beat you.
So each line of code is precious. And you don’t want to “waste” it on a test.
So I still wasn’t interested in testing.
When each feature was completed I manually tested it, then my co-founder tested it. Then I fixed any bugs. Then we both tested it again. Then, when it worked, I moved on to the next feature.
Then things that had worked would break. So I’d go back and make them work, then move on. Then, later, they broke again. A mantra began in my head, quietly at first, then with increasing volume: “Write some tests, write some tests…” But where to find the time with all this bug fixing to do….
Yes, that way madness lies.
Unit-ed we stand
So I began writing tests. This was a baptism of fire as, of course, there was now a large backlog of untested functionality to tackle. But I gritted my teeth, girded my loins (whatever that involves…), dove in and began writing unit tests. And, you know, a funny thing happened…
Slowly, very slowly, assertion by assertion, I learned to love testing.
I began to take actual pleasure (pleasure! imagine that!) in crafting a test then seeing the little green icon in my IDE ping to life when it passed. Knowing that my new feature was fit to go live. And more importantly, that I hadn’t broken something else in the process.
So, this was job done right? Sure, this didn’t test the front-end. But that’s what humans are for right?
Wrong.
The extra confidence, and speed, I gained from the unit tests simply seemed to manifest itself in more front-end bugs. Dammit!
Seleniummmm…
Then I discovered Selenium.
This was Selenium 1 (aka Selenium RC) so not the most stable and robust framework ever. We would often see a test fail then pass immediately after without any change to the code or data in between. Hmmm……
Even so, automating browser tests seemed like magic. It was mesmerising to watch the tests running. A never-tiring invisible hand filling in form fields and clicking buttons. Ok, maybe I’m just easily mesmerised.
It was fortunate that I found watching the tests so entertaining, because boy were they s-l-o-w. A full run would take around 90 minutes. It also sent the CPU fan on my laptop crazy and made using it for any other purpose at the same time a painful ordeal.
The upshot of this was that I didn’t run the selenium tests very often. Which in turn meant that they grew more and more out of date. Which in turn meant that I was even less likely to run them….
So we ended up falling back on manual browser testing again. Doh!
Hot Sauce
But, wait, what’s that sound? Enter stage left, our hero on a white horse….. it’s Saucelabs!
Yes, I remember distinctly the day I came across the Saucelabs website. I instantly Skyped my co-founder “Praise the Lord!” I exclaimed, or words to that effect. “Selenium is re-born!”
And for us it really was.
After a very small amount of painless integration, there they were, our browser tests running in the cloud. Sweet!
The test sweet…ahem suite still took a long time to run but it was now fire and forget. Just kick off the tests and get on with my normal business.
Although I did kinda miss being able to fry eggs on my Macbook when the tests were running locally. Those were damn good eggs!
Almost as big a deal as being able to run our tests in the cloud was that we now had a complete record of every Selenium test run, including a video of the test running!
Add to this the ability to do ad-hoc cross browser/OS testing with Sauce Launcher, and test against our local dev build with Sauce Connect and it’s fair to say we were pretty ecstatic!
This was by far the best situation we’d been in test-wise and the quality of the code showed it.
Carpe Tddium
But, you know, I’m hard to please.
My two biggest remaining niggles for me were the speed of the Selenium test suite execution, and the fact that our Unit and Selenium test results weren’t integrated.
I’d come to accept these limitations, until…
What’s this, the rumble of horses hooves again? Here comes the second hero of our piece, Tddium, charging into the fray!
The claim of Tddium to completely lift the testing burden – unit and selenium – into the cloud, and run both in parallel blew me away. To the extent that I was dubious as to how well it would work.
The answer…. very well indeed!
Now add in Tddium’s Github integration and Continuous Integration support and…. wow… just….wow.
I am currently running 8 Tddium workers in parallel and the runtime for my complete suite of unit and Selenium tests is down from around 2 hours to 15 minutes.
This has been a game-changer in my development routine. I’m now much more ready take risks and try stuff, knowing I can get such quick and comprehensive test feedback.
Continuous Inspiration
So yes, my conversion, is now complete. From throwing code over the wall to ‘those tester people’, to being forced by necessity to slowly embrace automated testing, to now realising the full potential of automation with Sauce and Tddium, it’s been quite a ride. And it’s not finished yet…
I still know that, someone, somewhere in another startup is still writing that line of code, and they might have the same idea as me. But now I’m not worried that they’re going to launch their feature before us. And they’re not going to beat us.
Unless….they’re using Sauce and Tddium too….
Dammit!
Tddium for JRuby
Posted: September 12, 2012 Filed under: Feature, Tddium Leave a comment »Here at Solano we’ve run over 13 million tests since Tddium launched. We hear on a regular basis that a fast, automatically managed continuous integration platform changes the way our users develop software. We’ve also heard from folks that want the power of on-demand testing and CI with Tddium but use JRuby. We’re therefore pleased to announce private beta for JRuby on Tddium. You can now run your tests on JRuby in either Ruby 1.8 or Ruby 1.9 mode, and together with all of the frameworks and backend services we support for MRI and REE.
Our JRuby support is in private beta today, and will be rolled out to all of users in the near future. If you’re interested in running your tests on JRuby, enter your information here or shoot us an email at info@tddium.com and we’ll get you set up as soon as we can.
Usability Enhancements to the Tddium CLI
Posted: May 14, 2012 Filed under: Tddium, Testing Leave a comment »We’re happy to announce some changes to the “tddium” command — the main CLI interface to Tddium.
To pick up the changes, “gem update tddium” to get version 1.4.1 or later.
Watch the video tour:
1. “tddium run” – Automatic Suite Setup and Testing
TL;DR: “tddium run” automatically creates a suite (setup for CI) for the current branch. No need to run “tddium suite” manually.
Tddium is built around the concept of test “suites” — the test files associated with a repo and a branch, along with other configuration data, so when we set out to build a CLI, we made the “tddium suite” command the first step a new user ran to create and configure a test suite, followed by “tddium spec” to start tests (back when we only supported RSpec). The “tddium suite” command is the one-stop suite setup and configuration utility — it creates new suites, and lets you edit suite settings.
We soon discovered that many users who follow the common topic-branch (or feature-branch, or git-flow) methodology had to go through the suite setup procedure often, sometimes many times a day. They would simply accept the defaults “tddium suite” automatically determined – for the test pattern used to select tests, the ruby version, and the CI origin URL, and then waiting for the suite update to persist in their Tddium git repo (“your git git repo is being prepped”) before starting tests.
We also found that new users were confused by the output and prompts from “tddium suite”. Much of the copy produced by “tddium suite” was written before www.tddium.com had much content, so it had to be both utility and HOWTO.
So, we renamed “tddium spec” to “tddium run”, and made it a whole lot smarter:
- Automatically creates a new suite (and configures it for CI!) or chooses an existing one with sensible defaults. To view or configure the suite, use the “tddium suite” command as before.
- Waits for your Tddium repo to be set up and automatically starts tests when it’s ready.
- Has better formatted warnings and status messages.
2. “tddium web” – Open the latest session in your browser
Instead of cutting and pasting a URL for a manual run from the CLI, you can use “tddium web” to automatically open your latest build in a browser.
3. Shared login across repos
Tddium used to require you to log on the CLI in once per repo – no more!
Now, your login is valid across all of your git repos.
Enjoy!
We’re busy working on more usability enhancements based on feedback from all of our great customers.
Don’t hesitate to send us questions, comments, or suggestions.
- The Tddium Team
Heroku Continuous Deployment
Posted: May 9, 2012 Filed under: Continuous Deployment, Heroku, Tddium 2 Comments »A few weeks ago, we rolled out preliminary support for automatic code coverage collection and custom post-build tasks.
Over the coming weeks, we’re rolling out better UIs in front of these features, but if you’re impatient, and you’re up for using our sample rake task, read on for end-to-end continuous deployment.
I’ll describe how we use post-build tasks and environment variables to implement continuous deployment of one of our own apps into Heroku, including running migrations.
Note: If you currently use Tddium’s push-on-pass functionality, this approach replaces it.
Step 1: Setup Environment Variables
The first step is to set ephemeral environment variables in Tddium containing sensitive parameters, like your Heroku app name and credentials.
$ tddium config:add account HEROKU_EMAIL my_heroku_login_email@example.com $ tddium config:add account HEROKU_API_KEY my_heroku_api_key $ tddium config:add account HEROKU_APP_NAME my_heroku_app_name
Tddium’s environment variables allow you to pass this sensitive information to your tests and the post-build hook that we’ll create – without having to check these in to your repository.
You can find your Heroku API key by logging in to your Heroku Account page.
Step 2: Install the Post Build Task
We’ve written up a sample post-build task that will push to Heroku automatically (gist). You can customize this task as you need. Over the next few weeks, we’ll be rolling out a more streamlined UI to make post-build configuration much simpler.
def cmd(c)
system c
end
namespace :tddium do
desc "post_build_hook"
task :post_build_hook do
# This build hook should only run after CI builds.
#
# There are other cases where we'd want to run something after every build,
# or only after manual builds.
return unless ENV["TDDIUM_MODE"] == "ci"
return unless ENV["TDDIUM_BUILD_STATUS"] == "passed"
dir = File.expand_path("~/.heroku/")
heroku_email = ENV["HEROKU_EMAIL"]
heroku_api_key = ENV["HEROKU_API_KEY"]
current_branch = `git symbolic-ref HEAD 2>/dev/null | cut -d"/" -f 3-`.strip
app_name = ENV["HEROKU_APP_NAME"]
push_target = "git@heroku.com:#{app_name}.git"
abort "invalid current branch" unless current_branch
FileUtils.mkdir_p(dir) or abort "Could not create #{dir}"
puts "Writing Heroku Credentials"
File.open(File.join(dir, "credentials"), "w") do |f|
f.write([heroku_email, heroku_api_key].join("\n"))
f.write("\n")
end
File.open(File.expand_path("~/.netrc"), "a+") do |f|
['api', 'code'].each do |host|
f.puts "machine #{host}.heroku.com"
f.puts " login #{heroku_email}"
f.puts " password #{heroku_api_key}"
end
end
puts "Pushing to Heroku: #{push_target}..."
cmd "git push #{push_target} HEAD:master --force" or abort "could not push to #{push_target}"
puts "Running Heroku Migrations..."
cmd "heroku run rake db:migrate --app #{app_name}" or abort "aborted migrations"
puts "Restarting Heroku..."
cmd "bundle exec heroku restart --app #{app_name}" or abort "aborted heroku restart"
end
end
Step 3: Authorize Tddium’s Worker Key
Run tddium account to get the Tddium worker key you need to authorize with Heroku.
Save the key in a file: tddium-worker-key.pub.
Then run heroku keys:add tddium-worker-key.pub
Step 4: Trigger A Build
That’s it! Push to your git repo to trigger Tddium CI, or trigger a build manually on your Tddium Dashboard.
When the push and migration completes, you’ll see a post_build_hook.log.
If you haven’t configured Tddium CI, read our getting started guide for more information.
If you don’t yet have a Tddium account, sign up now for a free trial!
Don’t hesitate to contact us at support@tddium.com for more information.
Update (10/25/2012):
Our awesome customers have pointed out a few gotchas and solutions:
- Make sure you have the ‘heroku’ gem in your Gemfile, or the above Heroku commands won’t work. We’ll soon be automatically including the heroku toolbelt in our workers, but until then…
- If you’re using Rails 3.1+ and the asset pipeline, make sure you enable the heroku user-env-compile labs feature.
Tests are Part of your Product
Posted: April 26, 2012 Filed under: Continuous Integration, Tddium, Testing Leave a comment »Check out the slides from my Railsconf 2012 Lightning Talk on Speakerdeck:
http://speakerdeck.com/u/tddium/p/tests-are-part-of-your-product-railsconf-2012
I’ll be expanding on these concepts and sharing my thoughts on how developer-written tests fit into a strong engineering culture over a series of blog posts in the next weeks. Stay tuned!
2 Million Tests!
Posted: March 3, 2012 Filed under: Tddium Leave a comment »I’m happy to announce that Tddium has just run it’s 2,000,000th test!
That represents well over 10,000 hours of test execution for rspec, cucumber, test::unit, spinach, turnip, and jasmine tests.
We’re also pleased to announce some great new integrations:
- Build notifications via HipChat, and a convenient new way to test notification integrations
- RabbitMQ
- Elastic Search
Stay tuned, we’ve got some exciting news coming over the next weeks and months!
Tddium Launches (with Riak and Spinach support)!
Posted: December 7, 2011 Filed under: Heroku, Tddium, Testing Leave a comment »We’re proud to announce that Tddium is open to the public! Read more in our first official press release.
If you’re not already a Tddium user, sign up for a free trial at www.tddium.com, or install the Heroku Add-On.
Even as we were putting the finishing touches on our launch, we added support for:
- Testing against live Riak databases. We now support live instances of most major DBs: MySQL, PostgreSQL, Memcache, Mongo, Redis, Riak, and SQLite3. Anyone out there using a CouchDB or Neo4J in their apps? We’d love to talk!
- Tests that use the brand new Spinach BDD framework. If you haven’t seen Spinach, we recommend checking it out.
We’ve also rolled out some new “beta” features:
- Bundle installing gems from private git repos
- Support for git submodules (both public and private)
- Support for fast_specs
Contact us if you’re up for beta testing!
We’ve got more great things in store for Tddium, so stay tuned!
Psych isn’t Syck
Posted: December 1, 2011 Filed under: Rails, Tddium, Testing | Tags: psych, ruby, yaml 2 Comments »Ruby 1.9.2 introduced a new YAML parser called psych, which in theory is better than syck, the existing parser. Syck is the parser in 1.8.7 and REE and still the default in 1.9.2 but it is unmaintained. Ideally, we’d be able to just switch over to psych with 1.9.2, but there are number of outstanding issues with psych: it doesn’t work with delayed job and it also breaks merge keys. Since psych isn’t quite ready for prime time, 1.9.2 defaults to syck unless libyaml is installed, in which case it will be compiled to use psych. The result is a confusion of rubies: you need to check the value of YAML::ENGINE to determine which parser your ruby has compiled-in — hardly a satisfying state of affairs. Worse yet, some very useful gems don’t work with syck, for instance VCR.
Until the syck vs psych confusion gets sorted out, Tddium supports two versions of ruby 1.9.2: the default with the syck YAML parser and an alternate with psych. To use the psych-enabled ruby 1.9.2 you can add the following to your config/tddium.yml and run tddium suite –edit to update the suite configuration:
# config/tddium.yml --- :tddium: :ruby_version: 'ruby-1.9.2-p290-psych'
And voila! your tests will run with a ruby that has psych compiled in.
Antilles: Testing CLIs that use Web Services
Posted: September 4, 2011 Filed under: REST, Tddium, Testing | Tags: api, aruba, cli, cucumber, rest Leave a comment »We’re happy to announce our first OSS release, the antilles gem!
Antilles is a companion to aruba that makes it easier to test CLIs that use web services by spawning a local stub HTTP server. It’s an alternative to fakeweb, for cases like aruba, where fakeweb can’t be used because the process being tested can’t have its network conveniently monkey-patched.
A major component of Tddium is the tddium command line interface. When we started building the tddium command, we wanted to write high-level tests, and we saw great examples in the cucumber tests written for the rspec command. These features use aruba to automate testing CLIs by setting up a sandbox directory and defining a number of convenient cucumber steps for running and interacting with commands.
We wanted a test like this:
@mimic
Feature: Login command
Scenario: Interactively log in successfully
Given the user can log in and gets API key "apikey"
When I run `tddium login` interactively
And I type "foo@example.com"
And I type "barbarbar"
And the console session ends
Then the output should contain:
"""
Logged in successfully
"""
And the exit status should be 0
And dotfiles should be updated
But we quickly realized we couldn’t do this with aruba alone. Our CLI relied heavily on our web service at tddium.com. In order to test properly, we’d need to stub out the web service. We’d started using fakeweb for other tests. But when we tried to make fakeweb work with cucumber/aruba tests, we found that the way aruba runs the command-under-test as a separate process, any network stubbing done in the test itself wouldn’t make any difference.
We had other things to worry about, like launching our service, so we chose what we knew – rspec – and postponed our thoughts of using aruba. Maybe our rspec-fu was lacking, but the rspec tests we produced were massive, complex, and convoluted. Our CLI uses Thor; each subcommand is an instance method. There’s 150+ lines of shared stub, fakeweb, and exit status detection superstructure. One 25 line method had 70 lines of rspec to cover all of its cases, on top of the 150 shared lines. These tests were complete, but hard to follow and even harder to extend.
In an ideal world, we’d have the command-under-test to talk to a “fakeweb server” running in a thread controlled by the cucumber/aruba process. Enter mimic, a stub HTTP server that can be forked and have its stub responses configured via an HTTP API. Antilles forks a sub process running a mimic server, and provides a simple API to hook it into Cucumber tests. Once the tddium command is configured to connect to the Antilles server instead of our internet-facing API server, voila, we have an end-to-end test that’s expressive and easy to maintain.
