ISO 29119 is a standard on software testing that was published in 2012/13 in 5 parts, and today has 8 published parts (the most recent being Part 13 - standards bodies are weird).
The standard attempts to codify the practice of all testing, as applies to all software. It defines vocabulary, processes, governance, management, techniques, and a lot of documents.
Having reviewed the supporting and opposing opinions from leaders in the industry (since I’m not actually gonna pay the £150-300 per chapter to read this in full), I have come to the conclusion that I do not support this standard, and stand firmly opposed to its usefulness, its applicability to software testing, and ideally its existence whatsoever.
The authors are the sellers of consultancy, training and certification programmes. The same people who wrote this standard wrote the ISTQB tester professional certification syllabus, an established racket for gatekeeping certain roles by duping the business decision makers that its responsible to have certified testers. The people writing these standards are again the same people selling training and certification in the standards - it’s a clear conflict of interest that gives us leave to doubt the motives of the working group pushing so hard to see industry adopt these standards.
They say the existence of the standard is harmless, since compliance is entirely voluntary, which of course it is… until it isn’t. If a government department decides that all health tech needs to be developed following “industry best practices” and so mandates that the standard be followed as a matter of policy, it then is immediately mandatory for all UK health tech.
They say it’s suitable for all sizes of organisation, big and small, and all styles of development, waterfall through agile, which it is… so long as you don’t want to be small and agile at the same time. The documentation burden alone, that mandates a written test plan for each project, test cases for each project, test outcomes for each project (none of which many organisations have a consumer for) would scale the level of effort required by testing enormously, as would the test management required to update said test plans as soon as any new risk is identified (since according to ISO 29119, all test risks must be identified in advance of the first test being performed).
Former auditor, James Christie, commented on the standard author’s assertion that following the standard gives business owners “peace of mind”, saying that confidence & peace of mind are valueless - you want the truth, gained through good testing, even if keeps you awake at night. The standard provides a framework in which to perform testing. You can still do terrible testing within it, and you can still do great testing without it. Michael Bolton (founder of Rapid Software Testing, a founder of the Context Driven Testing movement, and absolutely no relation to the singer) put it less delicately, saying that the standard “might be useful to people who are more interested in ass coverage than in test coverage”.
Part 1 of the standard deals in Concepts & Vocabulary, and having a common lexicon of testing makes sense, I guess. We can use terms like “pairwise determinism” and “equivalence partitioning” as terms, safe in the knowledge that the other party understands mostly what you mean.
Part 2 is around the Test Processes, and defines many levels of documentation to maintain, from Organisation Test Strategy through to Project Test Plans, which include identifying and documenting all of the risks before testing commences. Organisations keep pieces of text through Slack, Jira, Github, Confluence or Notion that serve the same purposes as many of the documents declared in the standard. What we don’t do is follow any pattern slavishly, wasting time documenting in advance (or at all) the 5 obvious tests required for a trivial bug fix, since it isn’t a good use of our limited time. Where our numbers are small and our ambition is large, dedicating what appears to be at least a third of our time to documentation feels like it would have a tremendous consequential opportunity cost.
Part 3 is about Documentation. Because the last one wasn’t? This deals largely with the “management” of testing. We’re unsure why the process appears to dedicate so much onus on our performance of test management rather than on the actual testing, wherein conformance to the process is monitored and enforced - we can only surmise it’s because you’re hiring rogues and chumps. This also covers the obvious and inevitable case of a new risk being identified once the software has been seen or used, and the re-authoring of the costly documents to account for it.
Part 4 covers Testing Techniques. There is a constrained list of 18 blessed techniques for testing. Some of them we’ve never heard of, but reckon we probably use most, if not all of these. Testers love giving boring things fancy words. Standards people do too. You can only imagine what happens when there’s a lovechild. We also believe we use techniques that are outside of this list, and variations of these techniques to achieve different effects. At no point do these techniques address observability, nor do they deal with outputs vs outcomes.
Iain McCowatt, Director of Quality at Barclays, makes an excellent point when he says that you can’t create a standard that makes something simpler whilst maintaining the original’s value and detail. Only something more complex can act as a control system. That’s costly.
All business managers understand the need for process, but also understand that blind rule following damages critical thinking, and onerous process is costly. Every decision we make is a trade-off, since time is limited and tests are infinite. Servicing the process instead of testing? If we’re dealing with process, we’re destroying value in testing.
In summary, I believe that:
It is important to increase the amount of knowledge, not documentation It is important to update skills, not documentation It is important to follow the level of quality of the product that is produced, and not the documentation that describes the theory of “how it could be done”
Testing is the art of finding something that did not raise questions at first glance, as are the methods of finding it. You cannot limit it within ISO 29119 rules and expect a good result.