Earlier this week I learned of a new Microsoft Research labs project called “Pex” (short for Program EXploration) that posits an interesting approach to crafting unit tests - more specifically, how to automate the coverage of the unit tests you may write.  For the uninitiated, “test coverage” is a moderately controversial topic in the Test-Driven-Development world that suggests unit tests should aim for 100% code coverage in order to be valuable as a regression validation tool.

 

At first, I was skeptical about Pex - I mean, after the debacle with VSTS back in '05 where MSDN docs actually suggested auto-generating unit test stubs thereby negating the whole point of changing developer habits to write tests first, I thought Pex was another run at the windmill from it's description:

 

As it turns out, this is a bit misleading;  what Pex is actually all about is a new style of unit test composition:  Parameterized unit tests.  This is interesting!

So, what is a parameterized unit test?
Take your run-of-the-mill unit test (this is from the site):

Nothing special here - just validating a hashtable.  Trouble is, we're baking-in some arbitrary values to validate our assertion.  What if we need to check several different sets of values for a more complex object?  We need to either: a) write more tests for each case, or b) invoke a bad practice and lump a lot of representative cases into a single test, or c) write a loop inside the test to go through several sets of values.  In other words, tedium which needs to be maintained as the system grows.

Now, take a look at the same unit test written as a Pex parameterized test:

Already, we see the first significant difference:  The test has parameters!  And these arguments are then fed into the test, suggesting by design that this unit test is going to be more flexible than its predecessor.  Pex doesn't stop here, though:  It is capable of generating the inputs for the test.  How?

Again, some interesting things are going on here:  Pex actually monitors the execution of the test by using random inputs based on the arguments.  Next, it uses a constraint solver to ensure that every set of inputs made actually exercise different code paths within the test and objects under test.  Thus, we have significantly improved code coverage.

This is a really radical approach to computer-assisted unit testing - Visual Studio will in effect become an intelligent aide to developers who will still be on the hook for writing tests, but relieved from the tedium of providing supporting code to ensure higher code coverage.

Pex is still under development, so we probably won't see this until after Orcas is released - too bad!  In the meantime, check out the site and have a look at the screencast which shows how this tool works within the VS2005 IDE - it's definitely raised my eyebrows.