http://en.wikipedia.org/wiki/Exploratory_testing
The actual definition of 'exploratory testing', is often lost, and is left to the imagination of individuals, and mostly misconstrued against the traditional way of testing, usually with disastrous consequences. I have seen some self proponents of Exploratory testing reduce the system test cycles, the emphasis on test planning and test case design, with an argument that Exploratory testing has found more bugs!
That clearly says whatever they followed was flawed. I have tested, and I have been trained to be systematic. If your test case design techniques are good, if you have a good traceability matrix, if you use code coverage early, then you are going to end up with test-cases that cover your product completely.
My theory has been, spend quality time creating testcases that cover (objectively define this) as much of classes, methods, branches and lines in the code. If you have done that, in the first few test runs itself you will uncover most bugs. Again, code coverage analysis helps *ONLY* covering all implemented code. The implemented code could still be wrong, so your test case should be RIGHT, and most importantly, some features may not be implemented at all. So your original testcase design through traceability will catch these scenarios.
So am I advocating completely against exploratory? Not quite. Exploratory testing is something I have carried out in a different way. After running test cycles through traditional system testing, if the bugs incoming rate is dipping, then we go exploratory, but in these structured ways.
- Scenario testing : We create an end-to-end scenario for common use-cases (the system test could be covering only one module, whereas a scenario test can walk through end-to-end. This is common when you are testing a huge product, such as any of Adobe products that we test :).
- Swap features : After the incoming bug rates dip, we swap the features and ask testers in the same team to test a new feature. A new eye, new perspective can sometimes find bugs. But use this late in the cycle, and SPARINGLY.
- Bugbash : After we are in a stable quality state, We invite testers from other teams, not related to our product and invite them to test for a day, of course, we return the favour to them, by testing theirs as well :) We hold these as a one-day event with prizes to be won.
- Public/Private Beta programs : This may not be applicable for all of you, but product companies do it always. We offer our product builds, either publicly, or through private invitation, (after signing a NDA), and invite our users to try and share feedback.
So my take is, Exploratory testing can complement and augment System testing, but can NOT replace it.
If you are of the opinion that Exploratory testing always finds more bugs than traditional, then it is indicative of these :
- That your traditional test case design technique is badly wrong - you have holes in your test coverage
- That you have started exploratory too early in the cycle, before system tests are completed on a relatively stable build.
Again, questions and feedback are most welcome!
You obviously don't understand what exploratory testing is or how to do it. I suggest that you spend some time reading what real proponents of exploratory testing have to say about it. Then learn how to do it correctly. Once you've done that you will discover, as I have, that it does find more defects than traditional testing methods. The main reason is that you spend more time testing and less time writing about what you're going to test. I'm sorry that you've had bad experiences with exploratory testing, but that doesn't mean you should give up on it. It cannot be done correctly by beginners. It cannot be done in a vacuum. You have to understand the product. You have to understand how to test effectively. Most importantly, you have to understand the user's needs. When you combine these three elements, and you create a few simple documents to guide you, your testing efforts reveal many more defects because you are able to put more time into testing. You can start testing earlier, and study after study shows that finding defects earlier in the SDLC saves money. You can test more effectively. If the requirement changes, an exploratory tester can start testing immediately without having waste time revising documents.
ReplyDeleteIf you want to learn, I suggest you read some of the references in the Wikipedia article into which you tried to insert this blog as a reference.
Thanks for your comments. As I said in the beginning of the article, I am not against Exploratory testing. I am against the mis-interpretation, timing of it and wrong following of that. I have been sitting through seminars and conferences, where people are walking away from formal test design and testing processes, and ending up with not testing enough, often missing code changes caused injections. If you read my thoughts, I have said, I did indeed follow exploratory testing in certain ways with decent benefits.
ReplyDeleteWalter - no dispute on early testing saves you money. I agree fully with that, in fact I am a proponent of static code analysis, if you read another paper I wrote on this blog, that even before we compile, we should find bugs.
Similarly, I have been writing and talking about gated builds, auto-BVTs, Pre-Integration testing and so forth. These are about catching bugs early.
And with exploratory tests, if the test cases are not created and maintained well, imagine the case of a product like ours, which lives for 15 years on an average, with code size ranging millions of lines, it is a nightmare to just live with exploratory testing with no structured test cases. How well can you guard against regressions and injections?
I am talking about testing in a larger scale over longer time with huge teams, with work spilling over newer and newer people. And believe me, we are not beginners :) If we are, we can not be shipping products that are among the most popular in the world.