Answer the following.
When you look at your existing set of regression tests do you:
a) shake your head in disappointment
b) run away in fear
c) scratch your head in confusion
d) all of the above
How many of you chose the bottom answer?
Regression Testing seems to be something that we all want to improve on but that we don’t do a lot about. As a result, we can have huge regression tests that take days to run and when it comes to maintaining them it is hard to know where to even start.
Now, there is a lot out there written about regression testing but not very much of it is very user-friendly and for anyone just beginning to look at regression testing it can be a bit scary.
There is nothing to be afraid of.
Regression Testing is running a set of tests (ideally automated) against an existing system to confirm that the introduction of new code/features does not affect existing functionality.
Let’s say you have a blog page. On this page you have your posts, a ‘follow’ button and a ‘share’ button and that’s it. Your blog is live and it was thoroughly tested before it went live so you know it works. Now, lets say you introduce a new feature; a search bar. Obviously, you thoroughly test the search bar and everything seems like its working just dandy. This is the point where you would run your regression tests to ensure that the introduction of the new ‘search bar’ feature hasn’t affected any of the existing functionality.
The creation of regression tests should start as soon as there is anything to test at all and should initially include all tests for any new or changed functionality. As more features and new functions are introduced to the software under test, more test cases to check new or rewritten code will be scripted. As a result a regression test suite can grow very rapidly. Regression test suites can become so big in such a short period of time that it can be nearly impossible to run manually. For this reason, regression tests should be automated.
One of the biggest problems that I have come across when dealing with regression testing is choosing test cases for inclusion. This is not always easy. In fact, it is probably never easy. I don’t know if there is a set way of choosing tests for inclusion in a regression test suite (I am yet to find anything that provides lots of detail on how it should be done) but this is how I would do it…the Bearded Tester’s ‘Pie of Regression’.
So, as you can see, the majority of test cases in the regression test suite should be for the most commonly used features and functionality.
This should be followed by ‘found and fixed’ defects. By this, I mean functional bugs that have already been found and subsequently fixed. Fixed defects seem to have a habit of popping back up and regressing once new code is introduced so I think this is a must have area for any regression pack.
Finally, the remaining pack should include the lower risk, complex functionality and less commonly used features.
It doesn’t stop here though. Regression test suites are continuously evolving sets of tests. Once the regression pack has been decided then it needs to be continuously maintained.
As new features and functions are introduced, the new test cases for the release should become part of the regression test suite to be ran each time a new version of the code is introduced. Similarly, as older functionality is replaced, older test cases may need to be removed in order to keep the regression suite both efficient and up to date. Essentially, as the code changes and evolves, the regression pack needs to adapt and reflect the application.
In order to be most efficient, as the regression pack grows, then it could be split into different functional areas. The regression test could then be ran in stages (instead of one giant ‘over-the-weekend test run) and limited subsets from the test suite could be ran individually to strategically target areas that have changed. The individual subsets could also then be used to perform targeted testing in order to build confidence in other areas of the application.
This is not necessarily the right way to do it but I don’t think its a bad strategy.
What do you think? How would you choose your regression pack? Is there a right way to do it? Are there other areas that you would include?