Rails: February 2007 Archives

(to try to make up for the previous rant)

I've been making big fundamental changes to ParseTree and consequently ruby2c's implementation and testing. As a result, I'm drowning in hundreds of failures (869 tests, 2217 assertions, 238 failures, 193 errors). Even though they run fairly fast, autotest isn't much help at this stage, so I resorted to (*gulp*) manual test runs, but with a small twist:

ruby test/test_type_checker.rb -n "/test_[a-b]/" | unit_diff

This lets me focus on one class, with only about 10 or so failures/errors at a time to keep the clutter down. As my favorite mgr used to say:

"Big alligators, Little alligators, Swamp."

With this technique, I work from the bottom up and slowly wade through and eventually drain the swamp.

Heckle is a mutation tester. It modifies your code and runs your tests to make sure they fail. The idea is that if code can be changed and your tests don't notice, either that code isn't being covered or it doesn't do anything.

Changes:

• 1 major enhancement:
  • Unified diffs for mutatated methods
• 4 minor enhancements:
  • Now returns exit status 1 if failed.
  • Added a simple report at the end.
  • Runs are now sorted by method.
  • Autodetects rails and changes test_pattern accordingly.
• 2 bug fixes:
  • Aborts when an unknown method is supplied.
  • Escapes slashes in random regexps.

From an IM with Eric Hodel:

"I ran it on [insert scary class name here]
it finally ran into a [method in the r's] about 2 hours later
I think it found no errors
so heckle will tell you one of three things about your code:
its well tested,
its poorly tested,
and its so coupled that test failures must happen because its crap." (emphasis mine)

I have to agree. When I started heckle I was hoping for somehing better than rcov that would tell you that your code is poorly tested... But code so poorly engineered that changing anything as bound to make everything fall apart looks like well tested (read: heckle-proof) code in a sense. We both think that running just the unit tests that directly test the MUT (method under test) would help, but really it wouldn't help enough. I tend to think at that point that you don't have technology problems, you've got people problems. Technology can't help that very well.

Suggestions? How would you differentiate between "well tested" and "tightly coupled crap"?