Multiple Ruby Patchlevels and Bundler Versions
Posted: May 31, 2012 Filed under: Feature Leave a comment »Fine-grained dependency control is essential to a keeping yourself sane as a developer. A coarse example is when you are writing code on OSX and your teammate is coding on Linux. Your teammate complains to you that some of the tests fail, but they all pass for you. You spend several hours debugging and realize that the order of files returned by the filesystem is different.
Now, let’s consider the Ruby development environment. The three ubiquitous fixtures in modern Ruby applications nowadays are:
- Ruby (of course!) including rubygems
- Bundler (and a Gemfile)
- rake (and a Rakefile)
RVM, rbenv, and other tools (bundler 1.2.0.pre) have sprung into existence to manage the wide array of Ruby implementations and versions. Bundler, of course, itself arose to manage the wide array of gem dependencies that make ruby such a great environment to work in. Usually, the Gemfile lists a version of rake.
One of bundler’s advantages is the ability to ensure that fine-grained dependency versions don’t change (as in Gemfile.lock). The devil, as they say, is in the details: minor version changes in gems can come with significant changes in functionality. The same holds true for Ruby versions, and unfortunately even for ruby patchlevels. For example, ruby-1.9.2-p320 isn’t fully compatible with ruby-1.9.2-p290, at least not when it comes to constant namespacing.
The issue also applies to bundler itself. Even with a tool like RVM, changes in bundler need to be manually tracked, at least for basic app bootstrap – bundler can’t install itself. Thanks to all of the folks working so hard on bundler, it’s seen some change over the past few months. While many developers keep up with all of the changes, some are working in environments where they can’t (or simply don’t have the time) and they’ve come to rely on the quirks of the version they started with.
Ruby Patchlevels
We already support Ruby 1.8.7, 1.9.2, 1.9.3, and REE, but we had only been supporting a single patchlevel for each of them. We upgraded some of our users from 1.9.2-p290 to 1.9.2-p320, and quickly realized that it caused some trouble with factory_girl. While many of our users depend on 1.9.2-p290, some had come to depend on p320.
Well, I’m happy to say that we’ve recently rolled out support for multiple patchlevels of a given Ruby release, and we’ll institute a phased lifetime policy where we’ll gradually end-of-life older patches.
We currently support:
- ree-1.8.7-2011.03
- mri-1.8.7-p352
- mri-1.9.2-p290 and -p320
- mri-1.9.3-p194
You can request additional patchlevels to be supported by emailing us at support@tddium.com. We should have them available within a day or two.
You can explicitly set the ruby version for a Tddium suite by updating to tddium-1.4.4 and running:
$ tddium suite --edit --tool=ruby_version:ruby-1.9.2-p290
Bundler Versions
The “tddium run” and “tddium suite” commands will now honor the exact bundler version they detect in your environment.
You can also use the “tddium suite” command to set it:
$ tddium suite --edit --tool=bundler_version:1.2.0.pre
Enjoy!
We intend these changes to make Tddium more closely conform with your development environment. Stay tuned for more updates, and as always, don’t hesitate to email us with questions, comments, or suggestions.
- The Tddium Team
New Feature: Archiving Suites
Posted: May 16, 2012 Filed under: Feature Leave a comment »We’re happy to announce a newly released feature: You can can now archive old suites. This feature is especially convenient if you follow a topic-branch or git-flow methodology, where you may be creating several new branches every day.
You’ll find an archive button in your dashboard. Click it to hide the suite.
If you made a mistake and you want to unhide the suite, or you just want to see everything, you can toggle the filter:
Click “View All” to show hidden suites. They will be highlighted, and you then have the option to Unarchive them.
Usability Enhancements to the Tddium CLI
Posted: May 14, 2012 Filed under: Tddium, Testing Leave a comment »We’re happy to announce some changes to the “tddium” command — the main CLI interface to Tddium.
To pick up the changes, “gem update tddium” to get version 1.4.1 or later.
Watch the video tour:
1. “tddium run” – Automatic Suite Setup and Testing
TL;DR: “tddium run” automatically creates a suite (setup for CI) for the current branch. No need to run “tddium suite” manually.
Tddium is built around the concept of test “suites” — the test files associated with a repo and a branch, along with other configuration data, so when we set out to build a CLI, we made the “tddium suite” command the first step a new user ran to create and configure a test suite, followed by “tddium spec” to start tests (back when we only supported RSpec). The “tddium suite” command is the one-stop suite setup and configuration utility — it creates new suites, and lets you edit suite settings.
We soon discovered that many users who follow the common topic-branch (or feature-branch, or git-flow) methodology had to go through the suite setup procedure often, sometimes many times a day. They would simply accept the defaults “tddium suite” automatically determined – for the test pattern used to select tests, the ruby version, and the CI origin URL, and then waiting for the suite update to persist in their Tddium git repo (“your git git repo is being prepped”) before starting tests.
We also found that new users were confused by the output and prompts from “tddium suite”. Much of the copy produced by “tddium suite” was written before www.tddium.com had much content, so it had to be both utility and HOWTO.
So, we renamed “tddium spec” to “tddium run”, and made it a whole lot smarter:
- Automatically creates a new suite (and configures it for CI!) or chooses an existing one with sensible defaults. To view or configure the suite, use the “tddium suite” command as before.
- Waits for your Tddium repo to be set up and automatically starts tests when it’s ready.
- Has better formatted warnings and status messages.
2. “tddium web” – Open the latest session in your browser
Instead of cutting and pasting a URL for a manual run from the CLI, you can use “tddium web” to automatically open your latest build in a browser.
3. Shared login across repos
Tddium used to require you to log on the CLI in once per repo – no more!
Now, your login is valid across all of your git repos.
Enjoy!
We’re busy working on more usability enhancements based on feedback from all of our great customers.
Don’t hesitate to send us questions, comments, or suggestions.
- The Tddium Team
Heroku Continuous Deployment
Posted: May 9, 2012 Filed under: Continuous Deployment, Heroku, Tddium 1 Comment »A few weeks ago, we rolled out preliminary support for automatic code coverage collection and custom post-build tasks.
Over the coming weeks, we’re rolling out better UIs in front of these features, but if you’re impatient, and you’re up for using our sample rake task, read on for end-to-end continuous deployment.
I’ll describe how we use post-build tasks and environment variables to implement continuous deployment of one of our own apps into Heroku, including running migrations.
Note: If you currently use Tddium’s push-on-pass functionality, this approach replaces it.
Step 1: Setup Environment Variables
The first step is to set ephemeral environment variables in Tddium containing sensitive parameters, like your Heroku app name and credentials.
$ tddium config:add account HEROKU_EMAIL my_heroku_login_email@example.com $ tddium config:add account HEROKU_API_KEY my_heroku_api_key $ tddium config:add account HEROKU_APP_NAME my_heroku_app_name
Tddium’s environment variables allow you to pass this sensitive information to your tests and the post-build hook that we’ll create – without having to check these in to your repository.
You can find your Heroku API key by logging in to your Heroku Account page.
Step 2: Install the Post Build Task
We’ve written up a sample post-build task that will push to Heroku automatically (gist). You can customize this task as you need. Over the next few weeks, we’ll be rolling out a more streamlined UI to make post-build configuration much simpler.
namespace :tddium do
def cmd(c)
system c
end
desc "post_build_hook"
task :post_build_hook do
cmd "git reset --hard HEAD" or fail "could not reset git workspace"
dir = File.expand_path("~/.heroku/")
heroku_email = ENV["HEROKU_EMAIL"]
heroku_api_key = ENV["HEROKU_API_KEY"]
current_branch = `git symbolic-ref HEAD 2>/dev/null | cut -d"/" -f 3-`.strip
app_name = ENV["HEROKU_APP_NAME"]
push_target = "git@heroku.com:#{app_name}.git"
fail "invalid current branch" unless current_branch
FileUtils.mkdir_p(dir) or fail "Could not create #{dir}"
puts "Writing Heroku Credentials"
File.open(File.join(dir, "credentials"), "w") do |f|
f.write([heroku_email, heroku_api_key].join("\n"))
f.write("\n")
end
puts "Pushing to Heroku: #{push_target}..."
cmd "git push #{push_target} #{current_branch}:master --force" or fail "could not push to #{push_target}"
puts "Running Heroku Migrations..."
cmd "heroku run rake db:migrate --trace --app #{app_name}" or fail "failed to run migrations"
end
end
Step 3: Authorize Tddium’s Worker Key
Run tddium account to get the Tddium worker key you need to authorize with Heroku.
Save the key in a file: tddium-worker-key.pub.
Then run heroku keys:add tddium-worker-key.pub
Step 4: Trigger A Build
That’s it! Push to your git repo to trigger Tddium CI, or trigger a build manually on your Tddium Dashboard.
When the push and migration completes, you’ll see a post_build_hook.log.
If you haven’t configured Tddium CI, read our getting started guide for more information.
If you don’t yet have a Tddium account, sign up now for a free trial!
Don’t hesitate to contact us at support@tddium.com for more information.
Order-Dependent Test Suites
Posted: May 4, 2012 Filed under: Uncategorized Leave a comment »The other day I was helping a customer with a persnickety failure in his MiniTest specs when I came across the following gem in MiniTest:
##
# Call this at the top of your tests when you absolutely
# positively need to have ordered tests. In doing so, you're
# admitting that you suck and your tests are weak.
def self.i_suck_and_my_tests_are_order_dependent!
class << self
undef_method :test_order if method_defined? :test_order
define_method :test_order do :alpha end
end
end
As a test suite grows it is all too easy for what should be independent tests to inadvertently depend on one another. Frequently the root cause is state left behind in the database or a class definition or monkey patch visible at the top level. In an ideal world, we would give our test suites as much tender loving care as the rest of our code, but the reality is that order dependencies do happen. The goal should be to drive toward at least test scripts that commute if not individual test cases. In the meantime, a little public penance may be in order…
When Do You Commit?
Posted: April 30, 2012 Filed under: Git, Software Engineering, Testing Leave a comment »Git repositories hold a wealth of interesting metadata in addition to the code itself. The number, frequency, authorship, longevity, etc. of commits reveals a great deal about software and its development. Depending on the content of commits and commit messages you may be able to infer the life cycle of software defects: when are bugs introduced, how long do they last, how disruptive are they and so on. If the bugs are significant or interesting enough to warrant detailed analysis, we can investigate by hand. One of my favorite examples of such an investigation that spans over fifteen years of development by some of the best systems guys out there is the study of sleep and wakeup in Plan 9.
Closer to home, I wanted to know when we’re committing to our core repositories. Internally, we can correlate commits with bugs, deploys, and individual contributors. Eventually, I’d like to be able to answer questions such as “Is there a material difference in the quality of commits at 2 PM and 2 AM local time”? The first step, shown below, was to compute the smoothed histogram of commits over the last 90 days. In the first cut, we ignore weekends and holidays (we’re a start-up, after all!). I did smooth the histogram by replacing the number of commits in each bucket with the average of the three surrounding buckets. The results for a busy repository show that we don’t really do much between the time we get to bed around 6AM US east coast time and when the coffee starts to kick in mid-morning.
I don’t think there a lot of deep conclusions to draw here, but it does look as though we work in four hour episodes. And once in a while, we’re up to all hours making the service better for our customers.
Not everyone is a caffeine fiend, however. The following is a plot compute in the same way as before but for a somewhat larger development team keeping saner hours. I guess lunch is at 1 PM.
Once I’ve had a chance to clean up the script used to generate these graphs I’ll post it. I think it would be interesting to share the commit histograms across a range of applications, development teams, and organization sizes.
Tests are Part of your Product
Posted: April 26, 2012 Filed under: Continuous Integration, Tddium, Testing Leave a comment »Check out the slides from my Railsconf 2012 Lightning Talk on Speakerdeck:
http://speakerdeck.com/u/tddium/p/tests-are-part-of-your-product-railsconf-2012
I’ll be expanding on these concepts and sharing my thoughts on how developer-written tests fit into a strong engineering culture over a series of blog posts in the next weeks. Stay tuned!
Installing Atexit Handlers On Module Load Considered Harmful
Posted: April 25, 2012 Filed under: Testing | Tags: at_exit ruby Leave a comment »A common idiom in Ruby testing frameworks is the use of at_exit as a way to schedule interesting work. I am not a fan — it is an idiom that is ripe for abuse. The C library’s atexit(3) function that inspired Ruby’s at_exit function was originally intended to allow the registration of handlers to tear down and clean up after a program during normal termination. It isn’t the right place for significant computation. Cleanup handlers don’t alter the result of the program, they just tidy up after it. Running the majority of the work of a script out of the atexit handler violates this principle.
As a concrete example, take the Ruby test/unit framework. When the test-unit module is loaded, it defines a variety of classes including test cases, test suites, and the runner class. The runner is responsible for coordinating the actual execution of the test suite. So far, so good. However, the last line of the module is a bare function call that ultimately installs an at_exit handler that invokes the runner an has it run all of the discovered test cases. The problem is that this takes the control of when the runner is invoked out of the hands of the user. If test/unit gets loaded as a side-effect somewhere deep in a dependent library, you are screwed. There is no simple way to disable just the one atexit handler and test/unit does not expose an API for disabling the runner. Thus, the obvious monkey patch is dependent on the version of test/unit that is installed which means you have to chase internal details of test/unit.
In practice, the result is that your test suite starts running even when it shouldn’t. For instance, we recently saw one customer that upgraded to Shoulda 3.0 and suddenly had the test suite run whenever he ran migrations! Others have had their test suites double on them.
To be fair, this is not a problem unique to test/unit, nor is it the fault of Shoulda. The specific case was easily resolved by adding the test-unit gem to the Gemfile or switching to just requiring shoulda-matchers rather than shoulda itself. Still, it shoulda never have happened and if test/unit didn’t install atexit handlers unbidden, it wouldn’t have.
The moral of the story: installing atexit handlers automatically leads to nasty surprises.
Handling JSON POST bodies in your Ruby/Rack Application
Posted: April 23, 2012 Filed under: Uncategorized Leave a comment »Want to post largish JSON objects to your web service? Merely think that using POST parameters as a transport is a little ugly? If your web service is implemented as a Rack application, take a look at this handy Rack middleware: http://gist.github.com/981176. All it takes to use it with Sinatra is a simple use statement like the following and you can start posting JSON to your heart’s content.
module Application
class AppServer < Sinatra::Base
use Rack::JsonPostBody
end
end
Ruby Debuggers for 1.9.x
Posted: April 18, 2012 Filed under: Uncategorized | Tags: debugger, ruby, ruby-debug19 Leave a comment »I don’t often use a debugger with Ruby, but when I do, I use…ruby-debug19? The ruby-debug19 gem has been the most common debugger for ruby 1.9.x for some time. Sadly, it is no longer maintained. What to do? We’re moving to the aptly named debugger gem and we recommend you do, too. As it happens, the rails team recently made the switch, too.
