Let us examine the division of labor between QA and Developers/Architects, as we apply the “Developers own Quality” methodology. Do we need a QA team at all? J. What’s a QA engineer to do?
… quite a bit, as it turns out.
“Developers own Quality” simply prescribes that developers own the results of testing, and that their task is only complete once the code passes enough tests to prove that it works. However, this does not imply that developers DO all the testing.
In order to provide more details, let us split a release milestone (or sprint) into 3 phases – for the purposes of this discussion, where we focus on quality:
(1) Design, development and TESTING,
(2) QA: Quality ASSURANCE
(3) QC. Quality CONTROL
(1) During the first phase, architecture design and development, the focus, from a quality perspective, is on testing, with a goal of demonstrating that the product actually works, and meets the stated requirements in all aspects of functionality and performance – and — that it works with the rest of the new code that’s been created during the milestone.
The testing efforts are lead by the architects or developers, with the QA team heavily involved: brainstorming on test cases, building and configuring test harnesses, executing manual tests – it is a team effort.
A key ingredient to this effort is: Architects, developers and QA engineers must ALL contribute test cases. There is joint ownership of test cases – each group brings its own perspective: the developer knows what’s inside, and thus what may be fragile, or what factors may limit performance. A QA engineer brings years of experience in testing, methodology, and his/her flair at identifying potential problem areas.
Cooperation is also critical in building the test fixtures, and generating the data sets that will exercise the full scope of the product. Architects often build the first barebones test-bed to validate their prototype. This prototype test-bed is then enhanced, or rewritten, during the development and testing phase, typically by developers, who then transition it upon release to QA, along with the product code. The QA team subsequently takes ownership of the test fixtures and continues to refine them.
Typically, during the architecture, development and testing phase:
o Product code is written by architects and developers
o Everyone must generate test cases
o Test fixtures and test data are created by architects/developers for the first generation, and subsequently enhanced and ‘productized” by the QA team
o Tests are executed by the QA team.
Ideally, as the code stabilizes, QA automates the tests; and adds them to the daily build and/or make them conveniently available for developers to set up and run the tests on their own (thus saving time for themselves).
(2) During the second phase, the Quality Assurance phase, the QA team rounds out the testing, and ensures that ALL test scenarios have been exercised, and pass.
What should be tested in the first vs the second phase is largely a matter of judgment: In the first phase, we do just enough to prove that the code works while in the second phase, we ensure that the code has no errors.
One way of to better understand this it is to consider exit criteria of each phase:
The exit criterion of Phase 1 is that no Severity 1 or 2 bugs will be found in Phase 2.
The exit criterion of Phase 2 is that no Severity 3 bugs (or worse) will be found in QC or after release.
The ideal exit criterion of the QC phase is no Severity 4 bugs (or worse) will be found after release. As we have all experienced, in practice the product owner (product manager) decides when to ship the product, trading-off time, resources and the very last bits of quality.
One may partition Phase 1 vs Phase 2 efforts based on the environments in which the product will run (e.g. versions of browsers, operating systems, databases). You select a representative sample of environments to test in Phase 1, and you round out the effort by testing the remaining environments in Phase 2.
Another way of looking at work allocation is in terms of risk management: all risk should be eliminated in Phase 1. This translates into: all bugs found during Phase 2 should require a predictable – and small – amount of time to fix; plus there should only be a relatively small number of bugs found in Phase 2. This very important point goes against the engrained habit of some organizations where developers test the basic case, and leave the worst case scenario to be tested by QA. On the contrary, ALL the WORST test cases must be exercised in Phase 1, and made to pass. Leaving it to phase 2 is just delaying the inevitable.
(3) Phase 3 is Quality Control of the “release candidate” – and is typically run by the QA team only. During the QC phase, the complete product is tested from top to bottom – newly introduced features, as well as those from earlier releases.
The QC phase may be abbreviated in intermediate milestones, but it is a critical step before an official release.
Ideally, by the time you reach the final QC stage, all the tests have been automated (functional as well as performance), and the QC phase goes very fast J
The above is, in my experience, a typical distribution of tasks, yet by no means is it a prescription. On the contrary, it is best for each team (architects, developers and QA jointly) to self-organize – as recommended by Agile.
A self-organizing team will review the tasks of each milestone, and adapt to the circumstances. For example, nothing prevents developers from helping the QA team run tests at the end of the release when it’s crunch time. And there is nothing wrong with a QA engineer writing the code of a test harness (it is even recommended).
I cannot emphasize enough how the importance of taking the time upfront – as coding begins — to figure out the test cases, testing harnesses – as well as test data. Unless you have sophisticated enough tests, you will never know how solid your product is. And, the sooner you have this information (i.e. in the development phase) the faster you will deliver the product.
Finally, to further emphasize the importance of the testing environment, in my view the test code, as well as the test data, are part of the “product” on equal footing with the code that’s shipped to customers. Test programs are just as valuable to the company as the code that they test. Or said another way, source code, without the tools to validate its correctness, has little value to a company. As a consequence, equal attention needs to be placed on the creation, maintenance, update, and safekeeping of test code and test data, as is placed on customer-facing code
o generate test cases
o Test fixtures and test data are created by architects/developers for the first generation, and subsequently enhanced and ‘productized” by the QA team
o Tests are executed by the QA team.
Ideally, as the code stabilizes, QA automates the tests; and adds them to the daily build and/or make them conveniently available for developers to set up and run the tests on their own (thus saving time for themselves).
(2) During the second phase, the Quality Assurance phase, the QA team rounds out the testing, and ensures that ALL test scenarios have been exercised, and pass.
What should be tested in the first vs the second phase is largely a matter of judgment: In the first phase, we do just enough to prove that the code works while in the second phase, we ensure that the code has no errors.
One way of to better understand this it is to consider exit criteria of each phase:
The exit criterion of Phase 1 is that no Severity 1 or 2 bugs will be found in Phase 2.
The exit criterion of Phase 2 is that no Severity 3 bugs (or worse) will be found in QC or after release.
The ideal exit criterion of the QC phase is no Severity 4 bugs (or worse) will be found after release. As we have all experienced, in practice the product owner (product manager) decides when to ship the product, trading-off time, resources and the very last bits of quality.
One may partition Phase 1 vs Phase 2 efforts based on the environments in which the product will run (e.g. versions of browsers, operating systems, databases). You select a representative sample of environments to test in Phase 1, and you round out the effort by testing the remaining environments in Phase 2.
Another way of looking at work allocation is in terms of risk management: all risk should be eliminated in Phase 1. This translates into: all bugs found during Phase 2 should require a predictable – and small – amount of time to fix; plus there should only be a relatively small number of bugs found in Phase 2. This very important point goes against the engrained habit of some organizations where developers test the basic case, and leave the worst case scenario to be tested by QA. On the contrary, ALL the WORST test cases must be exercised in Phase 1, and made to pass. Leaving it to phase 2 is just delaying the inevitable.
(3) Phase 3 is Quality Control of the “release candidate” – and is typically run by the QA team only. During the QC phase, the complete product is tested from top to bottom – newly introduced features, as well as those from earlier releases.
The QC phase may be abbreviated in intermediate milestones, but it is a critical step before an official release.
Ideally, by the time you reach the final QC stage, all the tests have been automated (functional as well as performance), and the QC phase goes very fast J
The above is, in my experience, a typical distribution of tasks, yet by no means is it a prescription. On the contrary, it is best for each team (architects, developers and QA jointly) to self-organize – as recommended by Agile.
A self-organizing team will review the tasks of each milestone, and adapt to the circumstances. For example, nothing prevents developers from helping the QA team run tests at the end of the release when it’s crunch time. And there is nothing wrong with a QA engineer writing the code of a test harness (it is even recommended).
I cannot emphasize enough how the importance of taking the time upfront – as coding begins — to figure out the test cases, testing harnesses – as well as test data. Unless you have sophisticated enough tests, you will never know how solid your product is. And, the sooner you have this information (i.e. in the development phase) the faster you will deliver the product.
Finally, to further emphasize the importance of the testing environment, in my view the test code, as well as the test data, are part of the “product” on equal footing with the code that’s shipped to customers. Test programs are just as valuable to the company as the code that they test. Or said another way, source code, without the tools to validate its correctness, has little value to a company. As a consequence, equal attention needs to be placed on the creation, maintenance, update, and safekeeping of test code and test data, as is placed on customer-facing code.