While waiting on hold the with my satellite service provider to complain about the quality of the service I received I had to laugh at the recorded message that was repeated over and over. A friendly computerized voice would interrupt my elevator hold music every couple minutes to say “To ensure quality, this call may be recorded”. Each time I heard it, I laughed.

If only it were that easy! If only a simple recording about my complaints could ensure that the quality I received was improved. If only my recorded call about my satellite no longer working and the terrible service I received as a result, complete with a INR 400.00 charge for something the company didn’t do correctly the first time could ensure greater quality. And perhaps the recording of my call could ensure quality for everyone else in any other situation as well! Wow, that’s pretty powerful.
This message got me thinking about a statement saying that a person, process, action, or thing can ‘ensure quality’ is a lofty statement to make. Those are some big shoes to fill! Ensuring quality isn’t something that should be taken on lightly. For starters quality is a difficult thing to define and it is probably different for every person. Quality for me, may not cut it for you and vice versa. Second, unless we have control over every aspect of the thing we are trying to ensure quality for, if we can manipulate it and change it where it needs to at any point in time we really cannot ensure anything.
So when my satellite provider says that are going to use my recorded call to “ensure quality” they really aren’t. What they really mean is, they are going to review the tape of the conversation I had with one of their employees . They may listen for ways they can improve, listen for mistakes that were made, and listen for ways to change what they do. They may listen because they care about improving their customer satisfaction. They may listen because they want to use my recorded call as a training example, good or bad, for new employees. They may listen because they want to find ways to make more (or spend less) money. They are listening to my recorded call for many of the same reasons we test software.

We test to find ways we can improve, look for mistakes we have made, find ways to change what we do, find ways to improve our customers satisfaction, use what we find as training examples, good or bad, and find ways to make more (or spend less) money. We test to gather information about our product, about what we are doing, about how we are doing it, etc. We don’t test to ensure quality.

So the next time I have trouble again and call for support, I will once again laugh when I hear the recorded message. Although ‘ensuring quality’ isn’t the real reason why they are recording my call, I do get the point. To repeat “To ensure we improve, get better at what we do, make more money, train our employees, and keep you happy when we care to, this call may be recorded” is certainly wordy.  However, I hope when they say they are “ensuring quality”, they also understand they really aren’t.

Congratulations, you just found a bug in an office supply commerce application!  It appears that any time you submit an order with more than one line item; an “object reference not found” error is thrown.  Cool!  Here are the repro steps you logged in the bug report:

1. Create a new order.
2. Add Pencils to the new order.
3. Add Pens to the new order.
4. Select UPS for shipping method.
5. Select Credit Card for payment method.
6. Submit the new order.

Expected Result: No errors are thrown.

Actual Results: “Object reference not found” error is thrown.

Those are pretty solid repro steps.  Every time one performs them, the Actual Results occur.  Nevertheless, do you see any problems with the above repro steps?

Hmmmmm….

What if you depend on just the repro steps to understand the bug?

Does this bug only repro if pens and pencils are ordered, if the UPS shipping method is selected, if the Credit Card payment method is selected?

Those details capture what the tester did, but might they mislead?  Anytime you add specific data to your repro steps, you may be implying, to someone, that said data is required to repro the bug (e.g., perhaps the “pens” data in our DB is bad).  A more skilled tester may log the bug like this:

1. Create a new order.
2. Add at least two line items to the new order.
3. Select a shipping method.
4. Select a payment method.
5. Submit the new order.

Okay, that seems considerably better to me.  However, we can still improve it.  Is it reasonable to assume the programmers, testers, and business analysts on our project team know how to submit an order?  Do they already understand that in order to submit an order, one must specify shipping and payment methods?  Yes!

Thus, an even more skilled tester may end up with these repro steps:

1. Create a new order.
2. Add at least two line items to the new order.
3. Submit the new order.

There is nothing extra to cloud our interpretation.  The steps look so minimal and easy, anyone could do them!  Now that’s a set of repro steps we can be proud of.