A while ago, I talked about a chapter in an online book I saw about Fast Build vs. Full Build. Well today, Jon Tirsen?talked about that same topic.

I think this is a very cool idea, but I want to take it a step farther. I think it would be awesome to automate this process.

Here's what I'm thinking. The build system maintains a list of tests that are part of the project and some extra data about each test: When it was created, when it was last ran by the fast build, when it last failed (if ever), and how long it last took. Then, for the fast build, you define an acceptable number of tests and/or an acceptable time for the tests to run. The build system would then be responsible for picking tests that should be run for the fast build.

Here's the tricky part. How do you decide which tests to run? Well, you'll get updated information about each test during the full build, which would be run every night or a couple of times per day, so you'll have updated information about any failed tests and the time they takes. Then, as the fast build runs, you get updated information for the selected tests. Then, as each build takes place, the build system picks all tests that have failed in the last x days. The number of days would have to be figured out, but the idea is that if a test has failed recently, you continue to include it in the build process for a while, regardless of other parameters. Then, if you're under your specs (test time or number of tests), then other tests will be picked based on three pieces of data, in the following order: Test age, so that newer tests are added. Test timing, favoring shorter tests,?and lastly, how long it's been since the test was last part of the fast build.

With the above in place, the build process could dynamically change as project focus shifts, because as soon as someone breaks something, that test will be automatically added to the fast build, and it'll stay there until it's fixed.

There is one potential issue with this approach that I can see. If a test fails that's directly related to what you're working on, you may not know about it until the full build runs. After that, it would get added to the fast build, but if you only do full builds once per day, then it could be 24 hours before you know about the problem, where as if you were picking tests by hand, then you may know to add that test to the fast build.

A way to take this a step further is to have a really smart build process that looks at two more factors: The code that is being modified and the code each test exercises. If the two coincide, then that test would be incorporated into the fast build process. But then you start to get into issues with how complicated the build process is, and it could possibly take longer to choose the tests than it would be to actually run the tests, which defeats the point of a fast build.

Maybe I'll mess around with something like this. It'd be a good chance to dig into the NUnit code to figure out how tests are identified and what I can do with that information.