Functional Test Matrix, cont. (sorta)

I've made some really good progress with the Functional Test Matrix idea. The code compression ratio is phenomenal. I'm writing up a comparison of the two, but without finesse the post would be huge and boring. I'll summarize here as best as I can.

I think most important idea is that high-level reviews or even edits of the functional tests require reading 1 line per action instead of N average lines per action * M edge cases. Very rough estimates show that my traditional functional test writing style weighs in at 179 lines of test for two actions against the following matrix:

setups         :u_w,   :u_r,  :g_w,  :g_r,  :o_w,  :o_W,  :o_r,  :o_R
matrix :edit, :edit,  :edit, :edit, :edit, :e_RO, :e_RO, :e_RO, :e_RO
matrix :view, :view,  :view, :view, :view, :view, :view, :view, :e_NF

Compare that to the 3 lines above and an additional 61 lines of "legos" used to put the matrix together (and, if you want, another 17 lines for the matrix framework itself). The liberal estimate puts that at 2.8:1, and the conservative estimate at 2.2:1. I suspect these numbers to be a bit high and for it to probably move down closer to 2.0:1.

The kicker for me tho, is that once the testing matures, I think we actually settle on the ideal of only really working with the matrix itself, pushing us up to nearly a 60:1 ratio! Like I said, this is rough, so YMMV. Look above. When my client comes to me and says "I think it'd be more secure all read-only errors were not-found errors", I wind up modifying only 4 result codes. If we change our permissions model and add unreadable and unwritable to group, I add 4 result codes and get 4 more tests for free.

A side benefit (from my perspective) is that the functional test matrix itself is high-level enough that it can be read and understood by the business owner or domain analyst. Allowing them to glance at it and immediately point out either missing edge cases or incorrect results.