RubySadism: February 2007 Archives

Heckle is a mutation tester. It modifies your code and runs your tests to make sure they fail. The idea is that if code can be changed and your tests don't notice, either that code isn't being covered or it doesn't do anything.

Changes:

• 1 major enhancement:
  • Unified diffs for mutatated methods
• 4 minor enhancements:
  • Now returns exit status 1 if failed.
  • Added a simple report at the end.
  • Runs are now sorted by method.
  • Autodetects rails and changes test_pattern accordingly.
• 2 bug fixes:
  • Aborts when an unknown method is supplied.
  • Escapes slashes in random regexps.

From an IM with Eric Hodel:

"I ran it on [insert scary class name here]
it finally ran into a [method in the r's] about 2 hours later
I think it found no errors
so heckle will tell you one of three things about your code:
its well tested,
its poorly tested,
and its so coupled that test failures must happen because its crap." (emphasis mine)

I have to agree. When I started heckle I was hoping for somehing better than rcov that would tell you that your code is poorly tested... But code so poorly engineered that changing anything as bound to make everything fall apart looks like well tested (read: heckle-proof) code in a sense. We both think that running just the unit tests that directly test the MUT (method under test) would help, but really it wouldn't help enough. I tend to think at that point that you don't have technology problems, you've got people problems. Technology can't help that very well.

Suggestions? How would you differentiate between "well tested" and "tightly coupled crap"?