I’ve seen highly effective quality teams and quite ineffective quality teams and noted vast differences in the processes, culture, and mindset of how these teams operated. While not a thorough analysis across the industry (working with two data points here), the reasons why the one team was ineffective seems very logical. Thus, it’s worth talking about it.
The number one goal of QA is to find bugs. You could say that it is to ensure quality of a product, and while that may be true let’s put that to the side because I have found that message to be an excuse that QA gives when they aren’t finding as many bugs as they should. Thus, using number of bugs found as a metric, you should be able to measure QA effectiveness. These are the key ways that I believe make QA most effective : 1) Quick code base turnaround and bug resolution, 2) Lots of interaction and questions with the engineering team, 3) Good common sense when testing rather than following elaborately detailed test plans, 4) Engineering should not provide step-by-step test instructions, 5) Good balance between automated testing and ad hoc testing, and 6) not every bug or feature ticket requires QA testing.
1) Quick code base turnaround and bug resolution
I had an experience with the highly effective quality team that I will never forget. QA logged a bug on a feature that I just completed earlier that day. I knew exactly what the issue was when I saw the bug and fixed it and tested it within 5 minutes. Checked in the change and resolved the bug. Our system was set up for QA to immediately pull the new change, who then tested the fix and found that it still was not working. The QA engineer reopened the JIRA ticket and assigned it back to me. Duh, I had my engineering blinders on and missed something obvious (this is the whole point of having QA – to find those bugs that engineers sometimes are blind toward). I fixed it correctly this time and sent it back to QA. They tested it and closed the JIRA ticket. This whole process occurred in less than 20 minutes. In other companies with much worse processes, who operate on dev branches that are only pushed to QA once per week, this whole process could have taken half a month, and the overall time consumed would be *much* greater than 20 minutes due to the context switching of both the developer and QA. For a highly effective QA team, they need to be able to work off the same code base as the engineers.
QA needs to understand how features work and anytime they have any doubt, the development team is there to help. The key, however, is for QA to ask lots of questions. The use of axioms is the documented explanation of features between engineering and QA. On the reverse side, engineering needs to ensure they respond and are available to QA when they need assistance.
3) Common Sense > Elaborate Test Plan
I recently looked into our QA’s test documents and they are filled with redundant, mind-numbingly obvious test steps. As part of testing feature X on the customer website, each test portion of Feature X was filled with the same two starting steps, “test step 1), log into the customer site”, “expected result: the customer site shall appear”, “test step 2), click on feature X”, “expected result: this should open the feature X page”. Seriously, I couldn’t believe it. It’s as if there is a mindset that test steps should be written for people with zero knowledge of the product. If that were the case, we should outsource testing to a much cheaper QA team. Then, can you imagine if Feature X changes slightly such that it is a sub feature of Feature Y, then all the test steps on how to navigate to Feature X have to change. This is inefficient and does not make for an enjoyable work experience for the testers. Any decent QA engineer who has basic knowledge of the feature will know how to test it (and if they don’t, they should seek help). Actually, taking this a step further for this specific example regarding the customer website, *all* features on any user-facing page should be self evident as to their purpose, and especially so for the tester whose job it is to test the website. If the features are not obvious, then you should really look into improving the website and the QA engineer should note this non-obvious feature. I don’t want to get too sidetracked on the specific case of a user-facing feature, the overall point: enable your QA engineers with the tools and knowledge they need to properly test the product and rely on them, not elaborately detailed test plans, to find bugs.