Testing the Untested

Test driven development is the cool new way to write software. TDD revolves around writing the tests for the software before or during the development of the software. If you do anything like that, then you'll end up with well tested software.

One of the fantastic things about well tested software is that you can change it confidently. Rapid change is the mantra of a large number of the newer software development methods.

I'd like to talk about the other end of the spectrum, though.

What happens if you've got a collection of code that doesn't have a good collection of tests? What happens if the code was written without thinking about testability?

It can become extremely hard to add tests to code like this after the fact.

I carefully phrased the problem so that you don't have to admit to writing the code yourself. I've written code like this, though. I'm guilty.

I've also recovered successfully from poorly tested code. What does it take to make this recovery?

Unit tests

Unit tests drive individual functions within a specific module. A good set of unit tests will help improve your API's reliability.

If you wrote the module with testing in mind, then you'd have a small executable that would load the module and run the tests. This could be done fairly quickly, so you could incorporate the testing into your build process.

If you didn't write the module to be tested, then there's a chance that the module depends heavily on the application's setup and shutdown routines. I hope that's not the case, because, if it is, you'll either have to refactor a lot of the setup code or you'll have to run your unit tests within the application.

The latter option means that, in order to run your unit tests, you've got to startup the entire application and then go into a mode where tests are run. Assuming the application startup time is significant, this will slow down unit testing and make it happen less often. Avoid this if it's possible.

If you can create a light weight process that can load the module to be tested, then you'll only be limited by the amount of time you can spend on tests.

Well... That's not quite true, actually.

In theory, you can write some unit tests and incorporate those tests into your build process, but you originally wrote that module in a corporate culture that didn't require unit tests.

I'll talk about that problem at the end of this post. It's important.

How Many Unit Tests Are Needed?

How many tests should be added right away? This is going to depend on the project's goals. You won't be able to get to the point where you can make changes with confidence until you have a mostly complete set of tests. Getting a complete set of tests will be a lot of work, however. The good thing is that you'll probably fix a bunch of bugs while you create the tests.

If you don't trust your code and you have an infinite amount of time, I'd recommend going all out and creating a complete set of unit tests. This will fix a bunch of problems with the code, and it will force you to understand the code a lot better. In the end, you'll have a lot more confidence in the code.

Of course, time constraints are likely to prevent you from spending that much time writing new unit tests. What's a more realistic amount of tests to add?

If you're adding tests late in development, it's probably because there's a problem that needs to be solved. If the problems come from a small number of modules, then these would be a great place to start.

Have a small number of modules with tests is still extremely useful. It will increase your confidence in those modules, and it will allow you to change those modules more easily.

System Testing

System tests use the entire application or almost the entire application to test a variety of use cases. Generally, these are black box tests. This means that you don't worry about what code you're testing. Instead, you test from the user's point of view. A system test will answer questions about what a user can and can't do.

System tests are often easier to insert into an application that wasn't designed for testing.

There are several ways you can create these.

First, you can use a tool that drives your application through it's UI. There are a variety of tools out there. They can click on buttons and menu items., I'm not familiar with any of them, so I won't make any recommendations.

The tools that I've seen strongly couple your tests to your user interface. This might be good if you want to test the details of your UI, but generally you'd rather have tests that are easy to maintain.

If you have a set of stable keyboard shortcuts, then you could use a tool like AutoIt or the Win32 keybd_event function to drive the program. I'd prefer this over something that looks for controls on the screen and sends mouse clicks to them, but I may be too conservative here.

A further improvement of this technique can be done by using a macro capability that's already built into the application.

This bypasses the user interface. You might consider that a problem, but a macro language is likely to be a lot more stable than the layout of menus and dialog boxes, so the tests themselves are likely to be significantly easier to maintain.

Besides sending in inputs, you'll also have to verify the program's output. Saved files, the clipboard, or scanning log outputs can all be used effectively.

System tests are significantly more complex to setup than unit tests. Furthermore, they take a lot longer to run.

Because they take so long to run, it is unlikely that you'll ever convince your team to run these as part of their build process. Instead, you're likely to have a separate machine or set of machines setup somewhere to run the tests. That's what we do here at PC-Doctor for unit tests.

It's easy to make system tests take so long to run that they become unwieldy. Try to focus your tests on individual use cases. Keep in mind the length of time that you'd like your tests to take, and keep that budget in mind.

Of course, you can always add more machines to your testing farm, but this might not happen until you convince your workplace that they're useful.

A Culture of Testing

A culture that thinks that unit tests are a waste of time isn't going to want to add tests to their build process. They aren't going to want you to spend the time to build the tests, either.

Changing corporate culture is a complex topic that will have to be in a different post. It's hard. I don't recommend it.

Instead, I'd hope that the project that you're adding testing needs it badly. In that case, the benefits of testing should be large enough that you may start to get people to notice.

Don't expect everyone to suddenly run all of your unit tests when they compile your modules. Instead, use your work to start a discussion about it.

Examples

I'd also love to give some examples. One project that I worked on many years ago went through the whole process. World of Warcraft desperately needs to go through the process. (It annoys me how few tests they've got.) I'd like to talk about both.

I'd also like to keep this post relatively short, however. I'll wait until part 2 to discuss specific examples.