Thursday, 13 February 2014

Exploratory testing is about exploration - unedited version of article for "Trapeze"

(That's what I initially sent to the journal. It got quite edited and rewritten on the way. I still like initial version better, so here it is.
Edited version can be found here:

Exploratory testing seems to become more and more of a a big thing lately, which fills my heart with joy, but in the recent years I realized that people mean completely different things when they talk about it. Sometimes this difference is amusing, sometimes it's enriching and eye-opening,  and sometimes it's annoying (e.g. when your colleagues refuse to try something new, claiming that they are already "doing this exploratory testing thing"). I don't claim to have an ultimate answer, of course, but I'd like to talk about what I mean under "exploratory testing", and why I love it so much.

For me, exploratory testing is all about the idea of exploration. It's not about using charts and session-based testing, it's not about agile environment, and it's definitely not about some list of heuristics (becoming the new "best practices") you absolutely must use. It's about asking questions, experimenting and gaining knowledge about the world (and the software under test in particular). And as a good explorer, of course you are also bound to keep good notes of your deeds. I like to think about it as if I were the Curiosity rover. Or even better, a crew member of the starship Enterprise (LLAP to all you fellow geeks out there): exploring brand new worlds and happily startrekking across the universe. Never knowing all of it, but having the tools, the desire and the attitude to acquire new knowledge.

As a tester, I have knowledge and assumptions about the software I'm about to test: business requirements, technology description, knowledge of the environment, common sense, etc.. I also have a map, based on my knowledge: test cases, test strategy, process I'm about to follow. And of course I have many tools to help me on my job: from special software to heuristics and ol' good test analysis techniques. All of that gives me a good place to start, but if I let it to define what I do, my job will become mechanical and it really wouldn't need the power that is human brain that much. When I'm doing exploratory testing (and I'm always doing exploratory testing), I have to keep asking questions and remember to readjust my assumptions. It absolutely blows my mind how much you can gain from such a simple idea!

First of all, it gives you efficiency, because by asking questions you gain understanding of software, you make sure you don't use outdated documentation, you get to know people on a team and what they can do, you are providing fast feedback, and you are helping everyone on the team to do better job with having more up-to-date information about the project than they would otherwise have. It also makes the job more fun, which you can't underestimate if you get bored as easily as I do. Another thing exploratory testing encourages you to do is to use techniques from other fields -  humans have been exploring the world since forever, and there is heaps of historical experience and wisdom waiting to be applied to testing software. Also, if you agree that testing is about asking questions and gaining knowledge, it can help you with the project roles that others try to enforce on you. I found that this is a common occurrence, that the product manager (or whoever is responsible for the application) presses the testing team into answering the "is it ready to go production", and "when it will be ready" questions. The problem here is that usually testing team don't really have the power to influence the situation much: they can't decide what is acceptable to be in production, they can't assign additional time for testing or for development teams, and they can't rush developers, designers, translators, etc. into doing their jobs. And if you have no power, you can't take the responsibilities.

All these seemingly abstract ideas form a perfectly practical approach, that I've been more or less successfully applying on my daily job for the last 6 years, even before I knew the terminology to talk about it and to understand exactly what I'm doing. I call that approach exploratory testing. Let me share with you its main points in a series of statements.

1. The mission of software testing isn't to "provide quality", it's to gather and provide helpful information to those who make decisions about the product and it's quality.
2. Exploratory testing is an approach that doesn't depend on SDLC, it can be applied to any situation (even outside of testing itself).
3. You can never know for sure how much time will it take to finish the testing. The challenges of planning and performing testing are rooted in the "work in the unknown" part. It's just like scientific research in that regard.
4. Exploratory testing consists of many iterations of two very different steps: discover and investigate. Each step has it's own challenges and goals. On the "discover" step you are concentrated on finding the issue (issue being a problem or a question), and on the "investigate" step your goal is to gain information you need to deal with that issue.
5. It is important to ask questions (no such thing as a stupid question!), to look at problem from different angle from time to time, to perform root cause analysis, to use new tests instead of old ones, and to keep notes.
6. Use automation to help with mundane tasks and free your precious time for smart tasks - not to replace manual testing of everything.
7. Give feedback as soon as you can make it useful. Give clear feedback, and make sure it isn't personal or insulting.
8. Prioritize your work (which features to test first, how much time to spend on a problem, which risks to mitigate and which tasks to do first, etc.).
9. Know your tools well: heuristics, practices, diagram notations, software that can help on the job, etc.
10. Know software you are testing from business and technological points of view: what problems is it supposed to solve, who are the stakeholders, and how does it work (on a system level at least).
11. Don't spend time on documentation no one will read or use.
12. Keep documentation that is in use up-to-date. Don't let your experience and knowledge of the project to stay only in your head.
13. Use test ideas to guide your testing rather than test cases to define it.

There is one more thing I feel necessary to say out loud. Exploratory testing (the way I understand it) doesn't require you to be an experienced tester in order to be good at it. A completely new to a product and/or software testing itself person can learn exploratory testing, use it right away and be awesome. The only thing that is absolutely required is the passion to explore. Everything else will come with the answers to the question you ask. I tested that with few new testers way back by teaching them to do exploratory instead of the classical "do step-by-step test cases until you learn them by heart" approach, and it worked, they are now totally awesome.

That's all, folks. Once again, I don't claim to know what exploratory testing is really about, but just one way of looking at it. I'd like to thank all testers, whose blogs I read for the last 8 years, because I definitely learned a lot by doing that. And specifically, the biggest thank you goes to James Bach and Michael Bolton for their absolutely brilliant "Rapid Software testing" course, which gave me ideas to think about, confidence that I'm not going the wrong way and most of all terminology I use to vocalize my thoughts on testing.

No comments:

Post a Comment