Weekend Testing Europe session: some thoughts on test reporting and tool evaluation

Nice group with a lot to share at today’s Weekend Testing Europe session, which focused nominally on test reporting but for me was an even better exercise in context evaluation and articulating my thought process. I’m an introvert who proceeds quite a bit by gut feel and I probably fall prey to all the common cognitive biases and maybe some uncommon ones. So it’s important for me to Show My (cognitive) Work just like I used to do in physics class.

I will soon be reporting out on a large group testing effort and I want to do this in a way that does not require me to maintain a tight link between “testing progress” and test case execution. Michael Bolton has written quite eloquently on this issue recently and I can’t say it any better. (One of the participants in today’s session suggested XMind for reporting and I will be looking at this tool this week.)

So that was the context I had in mind for today’s WTE session. I’ve done some tool evaluations for my job and I have come to trust my ability to get a sense, pretty quickly, of a tool’s fitness for a given purpose:

• Will the tool do the basics of what I have in mind out of the box?
• If not, could the tool be reconfigured to do what I need it to do? Would I need a developer to help me with that? (The words “will take development time” tend to go over like a lead balloon in my experience. And I can code a bit, but I’m a tester first.)
• If the tool looks like a contender, have I found any serious bugs in the first half hour of use? If so, that doesn’t bode well for my adopting it. 

If all of the above criteria pass, I would probably spend more time evaluating a trial version of the tool before I made a final recommendation.

The beauty of the WTEU session is that I had to explain all of that first to myself and then to the group. So THAT’S how I do that? Really? 

One last note: If I do not accept the tool for further study but, during my evaluation, I find a serious issue in the course of black box testing that I believe could cause harm of some sort to a user, I should report it to the company that produced the tool. Note that my main mission during the initial review, which should take 30-60 minutes at most, is NOT to find all the bugs I can in the application. It is to evaluate the tool’s fitness for my intended purpose.