Test Early

… does not only mean that tests must be run during development, but even more importantly, testing must start during the architecture design and prototyping phase.

As one does not wait until after the release to QA to start testing, by the same logic, one should not wait for the code to be complete to run tests – and make progress towards “proving that the code works”.

More specifically, performance must be validated during the design and prototyping phase. By the term “Performance”, I include individual server performance, scalability, fault-tolerance, longevity testing, error recovery, behavior under stress, etc. While it may not be possible to test everything with a prototype, one certainly has a duty to validate as much as can be tested. The sooner one tests, on the smallest code base as possible, the easier it is to (a) identify performance bottlenecks, (b) fix any issues and (c) minimize the impact of such fixes on the project and other team members. As we all know, performance shortcomings are among the most difficult problems to fix, and whose resolution time is hardest to predict.

In fact, one of the fundamental exit criteria of the architecture design and prototype phase must be that:  It validates Performance.

Another reason, in my experience, to test during the design phase is to engage the dialog on Performance between the Engineering team and the Business Owners (Product Management). In the abstract, we all want faster performance with every release. Yet, one has to wait until a first round of performance tests to see how close (or how far) we are from a given target. Thus the cost / benefit analysis of improved (or decreased) performance cannot start until the first round of test results. Only then, can the time and resources necessary to reach the desired level be evaluated with some degree of accuracy.

In some cases, “forcing” my team members to run performance tests is the only way to have them read the spec J. As they go through their design, I often remind them: “If you don’t know how to test it, you don’t know how to design it.”

Test Often

… the other half of the “Test early, test often” mantra reminds us that performance needs to be tested continuously through the development process. We have all experienced performance being impacted by the strangest things. The worst “death marches” that I have experienced were the consequence of a serious performance issue found in the last days of the release. I strongly recommend running a minimum set of performance tests within each milestone (or Sprint, if you use scrum methodology).

My “best practice” is to run performance tests continuously – from the first day until the last day of the project — on a dedicated system that tests the last stable release 24×7 (e.g. from the last milestone/sprint). The architects, and developers, will have coded some automated tests that exercise the corner cases of performance, and even stress tests. Furthermore, running the tests over long periods of time — 2 weeks minimum with the same executables — also tests against memory leaks and other resource exhaustion bugs.

Let us examine the consequences of “Developers Own Quality”.

Quality is already in the code at the time when it is delivered to the QA team

In other words, the code meets all functionality and performance objectives.  The obvious consequence – as suggested by Extreme Programming (XP), and Agile Software Development – is that, in addition to writing code, developers must also test it. More importantly, developers own the results of these tests.

Too often, I have heard developers claim that their task was complete once they had provided Unit Tests along with their code. Writing unit tests is a good thing, it is an important and necessary step, but it is far from sufficient. Rather, developers must take a results-oriented approach to testing, and ask themselves: do my tests PROVE that my code works?

Beyond a comprehensive suite of unit tests, which validate basic operation of the code, two main areas must be addressed: (a) integration and (b) performance.

Integration testing leads us to another XP and Agile best practice: frequent integration releases (or milestones) to ensure that all newly contributed code plays well together. For example, two developers will have often a different interpretation of an API. While each may have done the right thing in their own mind, and pass their individually created tests, the code, once integrated, will not work.

So, why ask developers, rather than QA, to test integration and performance? It is simply a matter of efficiency.

The process of releasing code to QA, having QA set up their test environments, find a bug, make sure it really is a bug, file a bug, assign the bug, re-run the test for the developer, wait for the fix, verify the fix, verify that the fix did not break anything else that worked before, and finally close the bug, is just too long a process. It should only occur in exceptional circumstances, or in controlled situations (more later).

To me it is also a matter of pride. As a developer, I need to be confident that I deliver solid work-product to my teammates. Finding a serious bug in my code (whether functional, or performance), once I have released it, should be a major embarrassment. I often tell my team – jokingly – “If QA finds a Severity 1 or 2 bug in your code, you owe me fifty bucks!”, as an illustration of the level of confidence and pride that one should have in one’s code.

In summary, comprehensive testing, is part and parcel of development. A developer who is proud of his/her code, and proves that it meets all functional, integration and performance requirements, is not only an efficient developer, but someone who makes his/her whole team efficient.