When to stop Testing -

There was this talk on Step-IN on when to stop testing - Point of Stopping Testing (POST). The speaker did a fabulous job in audience engagement, keeping everyone on the thinking mode. It was a new type of presentation where he extracted more thoughts from audience. Definitely I learnt few things today on presentation techniques. Thank you Shishank Gupta
http://www.stepinforum.org/STeP-IN_SUMMIT_2010/plenaries/Shishank_Gupta.html

I believe more could have been on the content; and he said it is still on the works. I went up on stage and added a new perspective for "More for Less", through code coverage.

I intend to add two of my methodologies, that worked and paid dividends for me in the past.

#1 : Code Coverage:
#2 : Through test results trending

#1 : Code Coverage:
For folks who haven't heard about this - Code Coverage is something you can turn 'ON', like a side process, when you do your testing, at no cost to your efforts or time (it may have 5-10% degradation on the performance, so it is advisable to turn code coverage OFF during performance testing alone), so after you are done with your testing, the code coverage tool tells you how much of application you have REALLY tested, which class, which method, which branch and which line. As a tester, you can now go and write test cases for uncovered regions (if you understand code and can create new tests, great, if not, talk to your developer buddy). And in many cases, more tests could be covering the same area of the code redundantly - get rid of them, it only adds extra baggage through all your test cycles and does NOTHING more. So, deploy code coverage as early as possible in the cycle, so you have a decent chance to reduce redundancy and add new relevant test cases. If you want to get to the depth of it, read my paper http://qualinfra.blogspot.com/2010/02/code-coverage.html

#2 : Through test results trending:

Remember the common results schema I talked about in Test Infrastructure? Now once you define that and have all test scripts, tools or even manual results transformed to this common format, it gets easy for you to get this kind of analysis and decision making power on WHEN TO TEST, WHERE TO FOCUS etc. I am linking a slide from one of my earlier slides below.
http://qualinfra.blogspot.com/2010/02/metrics-measurements.html - Slide 9.

These two methods combined have given me enormous confidence to decide on when to stop testing, or refocus my efforts where it is needed. Summarizing,

- with code coverage, we brought down the number of test cases (yes, that's a dangerous statement for a QA Manager, but with complete confidence, due to the measurements we have in place through coverage stats), and added new tests where relevant, overall, bringing down the execution time for a cycle, now multiply that with number of cycles, you get the picture on how to achieve more with less :)

- With the trend charting, we knew where exactly to focus and achieved more with less.

Of course, at Adobe we have a bigger tool to decide when to stop testing ... A prediction model that predicts and tracks for Zero Bug Count date, thats a far bigger discussion, I would continue in later posts.

Thanks for reading, and as always, buzz me if you have any questions.