Why I oppose adoption of ISO 29119

I don’t oppose the idea of a testing standard, though I’d like to see a programming standard to accompany it. But ISO 29119 and its predecessors are not testing standards. They are fundamentally standards for documentation of testing and things called “testing processes”. There is little that goes into a testing process practiced by a skilled tester that a document about documents can capture or codify.

In a long career I have yet to see any indication that so-called “test” standards have done anything to improve the skill levels of testers or the quality of their testing. Instead, I’ve seen many organizations doing mediocre rote testing with testers who are forced to produce reams of impenetrable, repetitive documents that nobody outside the company testing circle reads. I repeatedly see test strategy documents showing not an ounce of strategy yet compliant with standards such as this. Those same organizations often insist that their testers obtain certification.

Whether or not this is the intent of the ISO 29119 proponents, it is how adoption will play out in real life in many organizations. As James Christie has pointed out, contract lawyers, internal auditors and managers who know nothing about testing will insist on the grand panoply of fat documents because it’s a standard and therefore must represent “best practice”. Nervous and unskilled test managers will embrace templates based on ISO 29119 because all those documents make them feel secure and important. People on Agile projects will struggle with the conflicting demands of their projects and the standards.

I have yet to see evidence that compliance to any “testing” standard equates to good testing.

Testing is a skilled activity. (James Bach calls it a “performance art”.) The only true measurement of testing is skill exhibited in live practice. Some proponents of ISO 29119 sneer at the “craftsman” (or craftsperson) mentality espoused by many of us. I wrote in an earlier post that I grew up thinking of craft as “skill fuelled by love and integrity”. You who sneer at the idea of craft and make snide jokes about medieval guilds should take a look at some highly-skilled professions in the modern world. Do you think a surgeon never speaks of, nor works to grow, her craft? Is a person licensed to perform surgery because of the fine strategies, plans and reports he compiles in templates?

I don’t doubt that surgeons must plan and devise strategies. They have procedures they must follow and forms they must fill. But ultimately, surgeons are evaluated—and licensed by their state-sanctioned governing bodies—based on their results and the skill they exhibit on a real live human, tools of the trade in hand. They must also pass exams on their knowledge of the human body and its pathologies, as well as a range of tools and techniques. But the exams surgeons undergo are much more rigorous than anything developed so far for a testing certification. And no-one becomes a surgeon merely by passing exams. Like other craftspeople, surgeons serve an apprenticeship: studying, practicing and exhibiting on the job the skills they must have to qualify for  their profession. As do lawyers.

I’m not pretending that software testers normally require the same level of skill as surgeons, nor as extensive a education program. But I do think that scaled down the analogy holds.

I would welcome a real testing standard, though I’d like to see a programming standard to accompany it. A true testing standard would focus on demonstrated skills assessed by qualified practitioners. It might set boundaries for the levels of testing skill required to work alone or under supervision, and the types of software testers at differing levels could work on. Education to meet such a standard would combine classroom studies with on-the-job practical training, and judging of live testing. At successful conclusion of her education, a tester could be certified as a professional. Very skilled testers could become master testers, in demand for very high-risk software.

We aren’t nearly organized enough to devise a real testing standard in the near future. But I don’t see ISO 29119 as an acceptable substitute. It puts too much focus on the wrong things.