Good reasons for regression testing and why you need to think about it

How do you decide which existing tests to run when you make a change? The answer isn't that easy.

A test is at its most powerful when run the first time. After that its power melts like ice cream in the sun.

Yet, a lot of tests get run over and over and over again as part of some regression suite.

Why? They passed yesterday, why should today be different?

When I think about regression tests, they must fall into one of two categories:

Functionality that is so critical that you’d be fired for not testing it. Just in case!
Functionality that might have been broken by the changes that were made

The first category covers things like login, critical calculations or a basic level of testing on the absolute core part of the application. If you would lose your job for admitting that you didn’t test it before release, then it's in this bucket. This list stays the same, regardless of what is in a given release. It grows, slowly, as more functionality is added to the product. This category is based on reputational risk.

The second bucket is determined by the changes that have been made. This is where adjacent functionality, dependent functionality or functionality that for some reason always break when we make that sort of change goes. This is informed by the design of the system, historical experience and even developer or tester gut feeling. What’s in this category changes with every change. Determining what’s in this category takes knowledge, data and experience. This category is based on technical risk.

In a world where all the tests are automated, why don’t we just run the lot?

If you can, go for it. You are lucky, but you either have a simple application, too few tests or a massive test running budget.

If you aren’t so lucky, you have to choose what to run given the time and resources available.