The previous entry on continuous testing stirred up some discussion. Marko shows a script he wrote to save and test with vim , Nico dug into the rails setup, and found a way to have rake run only tests that changed in the last ten minutes (and then decides to refactor his blog while I’m writing, so I can’t quote him on that…) and I got an e-mail from June Kim with a very nifty solution to run tests continually, exactly at the time files have changed. I also found an earlier discussion on a testing list about using the ‘watch’ command, to look for changed but I can’t find the discussion back right now, so maybe more on that later.
I’ll comment on Marko’s solution first, and then I’ll discuss the (very nice) results I got with June’s script.
Marko’s script runs tests in vim, at the time of saving. Although this is very useful, I have hat in scite too (press f5 and the current file is ran), so this isn’t exactly what I’m looking for.
Based on a telephone conversation with Marko, I’ll try to refine the statement of what I want. I want tests to run ‘inside out’ Imagine a spiral, with the unit test for the bit of code you’re currently editing to be the focal point. Ideally, I’d want a test to run first for the method I changed last, then for the whole class, then for the suite the class is in, then further out to other dependencies. Tests run outward only when green bars are encountered. If there is a red bar somewhere, the spiraling stops, so we can examine the failure, fix it, and see again from the inside which tests run.
Usually, the further out in the spiral I am, the longer the tests take to run, so I want to run (and fix) the quick running tests inside the spiral first. If I don’t edit for a while, or go for a break, I want all the tests to run, including (at the end, when everything else is green) the integration/acceptance tests.
So, I like Marko’s script, it’s easy to modify, but it doesn’t run continuously, spiraling outwards. The other refinement in my wants, is that I want the run script to be independent of a specific editor. Everyone has their preferences. I can’t stand either vim or emacs. Actually, I have learnt to stand vim for a couple of lines at a time for systems administration tasks, but if I’m programming for a couple of hours, I want something that is modeless, has a mouse too and uses (keyboard) shortcuts I can actually understand (like ctrl-s for saving, and ctrl-mousewheel for zooming). If the script runs separately, any (one or pair) can work in the editor of their choosing, or whichever editor happens to be present at the time. Maybe, like Nico, I’m to young to enjoy working with vim.
I got an e-mail from June Kim, about a continous build script by Ward Cunningham for use with the IO Game :. It’s about seven lines of shell script, which is a bit more than the one line from the previous installment…
The extra lines are more than worth it, however! The script (like the last one) works on one file, checks for modifications in this file and runs the tests only if it is modified. The greatest thing about this one is, that if you save the file while the tests are running, it will kill the tests and start over. This setup seems excellent as a starting point for spiraling out testruns. Except for the fact, that I’m not really comforting with programming shell scripts (and this time for a change, I’m going to stay in my comfort zone), and some of my co-programmers use the windows os, for which the script doesn’t run (yes, it may run under cygwin).
Lucky for me, June has made an adaptation of this script in python . It is more lines, but for me much easier to modify. And, as a nice benefit, it tells us the time since the last testrun. This may tell us something about our courage…
The output looks like this, showing the current time, time of last run, the differences, and the time elapsed since the last run.
$ ./rerun.py sorting_grid_test.rb ruby
sh: line 0: kill: (12077) – No such process
[+] You changed…
previous Mon Dec 19 14:09:05 2005
+ current Mon Dec 19 17:28:27 2005
@ -3,7 +3,7
@
require ‘sorted_grid’
require ‘person’
-
+
class SimplePeopleGrid < UIBuilders::SortedGrid
def initialize
super
[+] 199m 22s elapsed since last run
[+] 14270 started
Loaded suite sorting_grid_test
Started
.FFF…..F
Finished in 1.782504 seconds.
1) Failure:
.... you know the drill
By the way, I got stuck writing this blog entry,because Python’s subprocess.Popen works differently on Linux (or cygwin) than under plain windows. If you want your run scripts to work on both, use the variant that takes a list of strings (e.g. subprocess.Popen([‘ruby’,’mytest.rb’) rather than subprocess.Popen(‘ruby myprocess.rb’)). I prefer to refer to code that has actually worked for me, and now it does .
Nico suggested, that having an all.rb file is duplication. I disagree. Rob and I use a simple script for a course that works with wildcards for files, so no need to change when files are added – it simply takes *test.rb in all subdirectories and runs those by requiring them. It works like this (we made it more fancy in reality, but you don’t have to do that):
!/usr/bin/env ruby
a couple of lines to make ‘require’ in subdirectories work
removed here…
require ‘test/unit’
Dir[’*/test.rb’].each {|t| require t}
The run script in Hourensou is longer, I don’t know why (yet) exactly.
I’m very happy with the results so far. Having tests run all the time, and seeing the changes is cool. I learnt a lot from pair programming with Rob, and from Marko’s, Nico’s and June’s responses.
In a next increment I hope to make a combination between June’s script and Nico’s suggestion to let rake do (part of) the work, or to extend June’s script so it watches all *.rb files in a directory. After that, I’d like to see how it goes for a directory tree, running from the leaves inwards.