Signing the ISO 29119 Petition and the Tester's Manifesto

I forgot to include these links in my previous post on ISO 29119. I have signed both. I urge all testers to read, consider the arguments and, if you agree, add your signature.

http://www.professionaltestersmanifesto.org/

http://www.commonsensetesting.org/news/files/PetitionISO29119.html

Why I oppose adoption of ISO 29119

I don’t oppose the idea of a testing standard, though I’d like to see a programming standard to accompany it. But ISO 29119 and its predecessors are not testing standards. They are fundamentally standards for documentation of testing and things called “testing processes”. There is little that goes into a testing process practiced by a skilled tester that a document about documents can capture or codify.

In a long career I have yet to see any indication that so-called “test” standards have done anything to improve the skill levels of testers or the quality of their testing. Instead, I’ve seen many organizations doing mediocre rote testing with testers who are forced to produce reams of impenetrable, repetitive documents that nobody outside the company testing circle reads. I repeatedly see test strategy documents showing not an ounce of strategy yet compliant with standards such as this. Those same organizations often insist that their testers obtain certification.

Whether or not this is the intent of the ISO 29119 proponents, it is how adoption will play out in real life in many organizations. As James Christie has pointed out, contract lawyers, internal auditors and managers who know nothing about testing will insist on the grand panoply of fat documents because it’s a standard and therefore must represent “best practice”. Nervous and unskilled test managers will embrace templates based on ISO 29119 because all those documents make them feel secure and important. People on Agile projects will struggle with the conflicting demands of their projects and the standards.

I have yet to see evidence that compliance to any “testing” standard equates to good testing.

Testing is a skilled activity. (James Bach calls it a “performance art”.) The only true measurement of testing is skill exhibited in live practice. Some proponents of ISO 29119 sneer at the “craftsman” (or craftsperson) mentality espoused by many of us. I wrote in an earlier post that I grew up thinking of craft as “skill fuelled by love and integrity”. You who sneer at the idea of craft and make snide jokes about medieval guilds should take a look at some highly-skilled professions in the modern world. Do you think a surgeon never speaks of, nor works to grow, her craft? Is a person licensed to perform surgery because of the fine strategies, plans and reports he compiles in templates?

I don’t doubt that surgeons must plan and devise strategies. They have procedures they must follow and forms they must fill. But ultimately, surgeons are evaluated—and licensed by their state-sanctioned governing bodies—based on their results and the skill they exhibit on a real live human, tools of the trade in hand. They must also pass exams on their knowledge of the human body and its pathologies, as well as a range of tools and techniques. But the exams surgeons undergo are much more rigorous than anything developed so far for a testing certification. And no-one becomes a surgeon merely by passing exams. Like other craftspeople, surgeons serve an apprenticeship: studying, practicing and exhibiting on the job the skills they must have to qualify for  their profession. As do lawyers.

I’m not pretending that software testers normally require the same level of skill as surgeons, nor as extensive a education program. But I do think that scaled down the analogy holds.

I would welcome a real testing standard, though I’d like to see a programming standard to accompany it. A true testing standard would focus on demonstrated skills assessed by qualified practitioners. It might set boundaries for the levels of testing skill required to work alone or under supervision, and the types of software testers at differing levels could work on. Education to meet such a standard would combine classroom studies with on-the-job practical training, and judging of live testing. At successful conclusion of her education, a tester could be certified as a professional. Very skilled testers could become master testers, in demand for very high-risk software.

We aren’t nearly organized enough to devise a real testing standard in the near future. But I don’t see ISO 29119 as an acceptable substitute. It puts too much focus on the wrong things.

Off the top of my head - Some skills & personal qualities that a tester can benefit from

My previous blog post - on why I believe it's good for testers to learn to code - triggered discussion and some protests, especially from testers who argued that testing involves so much more than an understanding of coding. Which is indubitable (to me at least).

So I thought I'd post this mindmap, an undoubtedly partial - in both senses of the word - list of tester skills and personal qualities that I threw together a few years ago in an idle moment. These are all things I believe a tester can benefit from. It's a very high-level, i.e., superficial view. I'm sure I've missed some very important items. Of the items I did list, it's clear to me that not all testers need every item in every context.
 

Why I believe it's good for testers to learn to code

Rob Lambert recently published a blog post with the title “Why Testers Really Should Learn to Code”. http://thesocialtester.co.uk/why-testers-really-should-learn-to-code Rob’s principal argument is that “The market demands it and the supply is arriving.”

What follows is an expanded version of the comments I made on that post.

First, Rob is presumably speaking about the market he knows best, i.e., in the UK. I’m not currently seeing such a heavy emphasis on coding in the Canadian market, though I think it’s probably there in Agile circles.

But regardless of the market demands, I think there’s a larger concern: about testers growing their skills and expanding their toolkits.

Whether or not testers “should” learn to code seems to be a contentious issue in at least some parts of the testing community at the moment. I admit that I’ve been observing the controversy with amazement. I’m having trouble understanding why any tester would not want to learn to code. 

I’m now a test manager, consultant and strategist, and I haven’t done serious hands-on testing in many years. But when I was a tester I knew how to code, and I worked at learning the languages I needed to know to understand and at least read the code for the applications I was working with. Working as a technical writer before I even became a tester, I learned to code and it seemed to me then to be an essential skill.

As coding advocates keep saying, you don’t have to be able to turn out production-level code. But it’s enormously helpful for a tester to understand how a system is constructed from the inside out. When you’ve tried to write working code, you learn to know the kinds of mistakes it’s easy to make with a given language and that helps you find bugs in other people’s code. And when you can read code, you can often spot the place where the error occurred and see what else might be wrong around it.

When you can code, you can write routines to build data in bulk, and also to inject data. You can write routines that help you test (or check) faster, or make it possible for you to test a larger number of input variations than you could practically manage otherwise. You can write and run your own batch jobs. You can query a database directly, to find out what’s really getting written to it. (SQL is code, too.) You can make clever use of spreadsheets to boost your test capabilities.

There are so many things a tester can do with code. And coding is FUN, folks! In fact, executing working code that you’ve written yourself is a blast! It’s almost as much fun as testing. (Okay, that’s highly subjective. But if you love software, why wouldn’t you love building some?)

There's a social aspect too. Being able to write code helps you understand your programmer teammates and it teaches you empathy and respect for their skills. (You want programmers to empathise with you and respect your skills too, don’t you?)

 

This is not to say that you can’t be a good tester without knowing how to code. Of course you can. I know lots of excellent testers who can’t code and don’t want to learn. I also know lots of terrific testers who don’t have (or don’t believe they have) exploratory testing skills, or visual thinking skills, and don’t want to learn those either.

In my experience, these and all your other skills give you tools you can use when the context calls for them. Not every tool is appropriate or useful in every context. But the more tools and skills you have at your disposal, the more flexible you can be and the more easily you can rise to the demands of different contexts. If you don’t have a particular skill, you may not even recognize how having it could help you test better in your context.

I don’t believe that the issue comes down to should or should not. Rather, I believe it’s about expanding your skills and your toolkit. Why wouldn’t you want to do that?

Breaking the Tyranny of Form - Part 1

Testing in many mainstream organizations is choked with low-value standardized documents that not only gobble up valuable thinking and testing time, but actively discourage thinking and impede good testing. While some testers can hope for relief from the document burden through the spread of Agile methods, this ridiculous situation isn't going away in a hurry. As a blog post by James Christie recently reminded us, the floodgates on “ISO/IEC 29119 Software Testing – the new international software testing standard” will soon open. I suppose it's possible that the new standard will wash away the documentation excesses we have now. I'm not holding my breath on that.

Test documents, whose sole purpose should be to serve the work, are instead driving and constraining the work. Absurdly, form is dictating substance. When, as in this case, the form is obese and bloated, it sucks up and squanders all the energy that ought to go into the real thing.

Some testers (many, I hope) refuse to be tyrannized by the supremacy of form. I want to reach into the mainstream where form dominates and help testers there to join us. I want to help them learn to think better and think for themselves. This blog post is a step in that ongoing effort.

Another step is the webinar I presented last week for EuroSTAR: "Unburdening Testing - Finding the Balance Point for Test Documentation". (The Q&A from that session are in blog form on the EuroSTAR site.) The webinar is an introduction to the interactive tutorial I will present November 6 at EuroSTAR 2012: "Right-sizing Test Documentation".

After presenting my own webinar, I watched the recorded version of Alan Richardson's excellent webinar "Thinking Visually in Software Testing". The thinking tools and practices he describes there are so uncannily like my own that it prompted me in writing this post to think and write about my thinking. (I encourage you to watch Alan's webinar, if you haven't already done so.) 

Form over Substance

When I first began working for a consulting company, my project manager gave me a mantra for billable work: “Never create anything that is not a deliverable to the customer”.

She was a brilliant PM who became an important mentor for me, but not all her advice was equally stellar. That statement in particular put horrible shackles on my work that I took years to shed completely.

The problem was that my customer deliverables were formal, standardized prose documents. For me to show value, I needed to end each day with sections and sub-sections populated with tidy paragraphs, building up to the finished product.

At first, I was lucky. My company was only dimly aware of standards purporting to govern test documentation, and so I created my own templates. But over time, my templates became our company standards. When I used them, I was so busy trying to adapt the structures for each project that I could no longer work easily with them. Like the templates for test documentation imposed in many companies, I found that the structure—the form—acted as a constraint on the substance. The tail was wagging the dog.

We Can't Test Without Thinking

Testing is all about thinking. We think and rethink constantly. We think about how best to gather information on our projects: what to read and look at, and who to talk to and when. We think about how to approach the software and we develop test ideas as we go. We create test models, often complementary models for testing different aspects of the same piece of software. We plan and replan and then plan again. We think about what we've discovered in testing and design what we're going to do next. If we're doing a good job, we don't ever stop thinking.

Prose documents in preset patterns inhibit thinking and creativity. It’s useful sometimes to have a checklist of important things to think about, but we can’t afford to let those checklists limit our thinking.Templates are not the best checklists.

Writing in sentences can sometimes help me to simplify and think through a tangled knot of ideas. I do occasionally write to understand what I’m thinking.  But I never set out to think in predetermined sections of formal standardized prose. Do you? Does anyone? Can anyone?

Visual Thinking

  A couple of years ago these drawings helped me think through a problem

Most often, I think in scribbles and doodles, beginning with notes and drawings that acquire structure as I develop the ideas and concepts. Or I might start with a tentative visual structure to generate ideas and then modify or replace the structure as needed to fit my thinking. Sometimes I scrawl ideas on coloured sticky notes and move them around over several days on a board or double-page spread of my notebook, drawing connections and annotating as I go. I often use mindmaps. I may use several different techniques to get my hands around a difficult problem. 

Diagrams Emerge

What comes out of my initial thinking processes is rarely a customer deliverable. But over time, the result is usually some kind of structured diagram or set of diagrams that I can then use to communicate my ideas to other people. Rather than dictating and constraining the substance, the form of these diagrams emerges from the substance.

Example of a strategy diagram for an end-to-end systems integration test on a large project

When—as is true on most projects—I must produce a formal document, I prefer to put diagrams front and centre. I want my documents to communicate, and I try to make them easy to read and understand. I use prose as sparingly as I can get away with, using tables and lists wherever possible.  I don't include boilerplate, and I never copy wodges of text from one document to another. (Why ever would I waste valuable project time on such useless make-work?)

This diagram shows the division of responsibility for testing on the same big project

Vacuous Form Tyrannizes the Mainstream

Apparently, that's not how most testers and test leads develop test documentation. In my consulting work with clients, I constantly see mammoth documents stuffed with books worth of low-content stodgy and opaque prose. Often, I search so-called test strategy documents in vain for any actual strategy. I fall asleep looking at test scripts that dictate every point and click and hideously repeat over and over again the most minute detail of so-called "test steps" and their piddling expected results. It's very hard to believe that much thinking is represented therein—or will inform the testing that must unfortunately follow.


Nobody reads this stuff. It isn't useful. It certainly doesn't help anyone test well. So why do testers waste time and spirit churning it out? Why do their managers insist on the tyranny of form over the substance that only thought can produce?

Let's Take Testing Back!

I believe that thinking testers must take testing back from the process weenies and form merchants.  Many testers have done this, but it has yet to happen in the mainstream—in big companies, big banks, government projects...sometimes even in startups and small companies. 

In subsequent posts on this topic, I'll explore some ways we can do this.

Let's Test 2012 - A Personal View

 Let's Test 2012

I'm rarely tempted to write about conferences, but for once I can't resist: the inaugural Let's Test was a wonderful experience in so many ways.

The Place

The rural setting at Runo, with its stunning "nobility of labour" art and bright, airy buildings, was an inspired choice, and no small contributor to the conference's success. We were all captive--miles from other (lesser) attractions--and there were many comfortable and inviting places to sit and confer, both indoors and out. It didn't hurt that there was a bar (with free drinks provided by sponsors on a couple of evenings), and fabulous food (the smoked salmon!). All meals were provided and the breaks were generously provisioned with treats like raspberry smoothies or pretty little macarons. When did you ever eat well, or even sit comfortably, at a conference?

Organizer Johan Jonasson, who knows I carry my own coffee to conferences and frequently joins me in a cup, promised me good Swedish coffee at Let's Test. It wasn't brilliant, but it was indeed decently strong and far more drinkable than I'd ever get in a North American hotel.

The Participants 

I use the word advisedly. The people at Let's Test were active participants throughout. Adopting the CAST facilitation model may have helped, and it certainly kept in-session discussions on track, but I don't think anything would have kept this group from lively and interesting conversations, often far into the night. And the Test Lab drew a big crowd every evening. That's what conferences are for!

The Sessions

People like me who present at a lot of conferences can become very jaded. At most, I rarely find the official sessions compelling and tend to skip them in favour of corridor conversations and networking. In contrast, the Let's Test agenda presented the best dilemma I can experience at a conference. In several time slots, I had difficulty deciding which session to go to. I was glad I chose Christin Wiedemann's "You Are a Scientist - Embracing the Scientific Method in Software Testing"  and Alan Richardson's "Testing Hypnotically", but sorry to miss several others. The real standout for me was Anne-Marie's Charrett's "Coaching Testers", especially the fascinating live session at the end, where many of us got to offer suggestions to (i.e., "coach") the coaches.

CAST is the only other test conference I know of that has this "assembly of peers" feeling, where speakers are as interested in other people's sessions as they are in their own.  

Presenting

I presented a full-day tutorial on Test Leadership and a track session on managing an end-to-end systems integration test.

I felt a little trepidation before the tutorial. How would a big group (28 people), for the most part meeting me and each other for the first time, and speaking what was for most a second language, work together in an experiential session? Would they embrace exercises that might seem strange to them, and then be willing to share their experiences and learnings in a debrief?

I needn't have worried. I started with a small-group exercise that probably helped to draw participants in, particularly some of the quieter ones. In that first exercise, one of the small groups--all Swedes--sensibly worked together in Swedish, though their output was in English. Otherwise (to my shame as a less than marginally bilingual North American), this multilingual group blew me away with their facility and willingness to communicate in my language. Soon everyone was engaged and working together, and they stayed engaged throughout the day (some more quietly than others), coming up with many fascinating insights and strategies for dealing with issues in test leadership.

I am grateful to everyone in my class for making it such a great learning experience for me! So far, I've had very positive feedback from participants. If you were there, please let me know what you think. Did the session work for you? What went well? What do you think I could do better next time?

Meanwhile, here's a link to Simon Morley's blog, where he describes his reactions.

Onwards (for me)!

I'll be keynoting (Are You Managing Test - or "The Test Process") at the BCS SIGIST in London on June 21, and also doing a workshop on mind-mapping a test strategy. I look forward to working with a group of engaged British testers there.

June 26-28, I will be presenting Beyond Process: three 1-day experiential classes in London, including a reprisal of my Let's Test tutorial, "Inspiring Testing" plus "Determining Business Risks for Testing" and "The Design behind the Plan - Test Strategy workshop". Registration is through the sponsor, ElectroMind.

If you're in North America, you may want to register for Beyond Process in San Jose before CAST, July 11-13.

Then it will be time for CAST 2012, July 16-18. As Program Chair, I believe the Program Committee and I have assembled an excellent program of sessions and innovative workshops. I'm really looking forward to seeing some of my new and old friends from Let's Test, as well as many others who didn't make it to Sweden. It too is going to be a wonderful conference!

Onwards (for Let's Test)!

Let's Test 2012 was a terrific first conference. My thanks and congratulations to the organizers, and I look forward eagerly to Let's Test 2013!

And some pics from the Nature Reserve

One of the evening activities was a Nature Walk, guided by a marvelously fit 80-year-old gentleman,  recently named Sweden's Gardener of the Year. It was so beautiful, I rose early the next morning and went out again with my camera.

Six Impossible Things Before Breakfast

As an accompaniment to my June 2011 Tea-time with Testers article, "Six Impossible Things Before Breakfast", here is the list of lies I have heard on software projects over the years. I’ve grouped them roughly; arguably there are some lies listed that cross categories. Also, there is some overlap. “We have to get creative about these numbers!” is one variation on falsifying progress and status.

I invite you to add lies you've heard to my list. I'd love to hear about your experiences and observations of software project lying, and also of truth-telling in difficult project circumstances. Please leave a comment.

As a reminder, here are the parameters I set for my list.

I had personally to have heard the lie told one or more times. Each lie or category of lies had to be material to a software project, though it could have been told to make a sale before a project began or to describe a project after its end. Each lie had to be relatively common in the industry—or at least not rare. A lie had also to be significant to a project: to have influenced perceptions and/or decisions. And finally, I excluded malicious lies intended to subvert or sabotage an individual on a project.

Lies told to make a sale or get funding for a project

1. Deliberate underbidding [but we’ll make it up with change requests]

2. Bait & switch [proposing highly skilled staff to make the sale, then replacing them on the project with junior and lower-skilled people]

3. “The system will be standalone.” [To be usable, this particular system actually required expensive integration with several major systems, both upstream and downstream. The development team told management this before the project was approved, but the project sponsor ignored their information. I’ve heard similar claims on other projects.]

Management lies

4. Fictitious schedule: fixing scope, cost and staffing after arbitrarily slashing informed estimates by the project teams

5. “Our software is bug-free.”

6. “We can cut the testing in half without affecting quality.”

7. “This project is life or death for the company. If it doesn’t succeed, we’ll all be out of a job!”

8. “We’re slipping the date out, but don’t tell the team. We need to keep their feet to the fire.”

9. “We have to get creative about these numbers!”

10. “We can solve all the project’s problems with mandatory overtime.”

11. “The project team is pulling out all the stops to deliver this project.” [Actually, they’re exhausted and making a lot of mistakes, and we’re only getting about 30 hours productivity from the 60-hour weeks we’re making them work.]

Lies about programming

12. Code complete. [On one project, the programmers had actually left comments in the code listing what had not yet been done.]

13. “We’re done!” [Though we haven’t done the unit testing we committed to.]

14. “I only changed one line of code. You don’t need to test it.” [Though I didn’t do an impact analysis of the fix and I have no idea what might break.]

15. Padding estimates

16. Hiding bugs

17. “It’s not technically possible to do it that way.” [It is, actually, but I want to use this cool new technology that will enhance my skills.]

Lies about testing

18. “Anyone can test. We can get people off the street to do this job.”

19. “Anyone can test. We just have to give them the right process to follow.”

20. Tester exaggerating risk

21. Tester padding estimates

22. “This bug is out of my scope, and I’m under the gun. I don’t need to report it.”

23. “Testing is holding up the project. You’re finding too many bugs!”

24. “Our test cases will provide complete system coverage.”

25. “The infrastructure upgrade will be completely transparent. You won’t need to test.” [Although nobody did an impact analysis of the upgrade and we have no idea what might break.]

26. “We can deem these test cases passed.” [Though they haven’t actually been executed for months and we strongly suspect many of them wouldn’t pass.]

27. “You only need three weeks for testing.” [Because the code is late and we cut three weeks from the test schedule.]

28. “The testers don’t know what they’re doing.” [They estimated it would take two weeks, but found incomplete code and so many bugs it has taken six, and they’re not done yet.]

29. “Our mature test process employs all the industry best practices.” [Major bugs frequently bring production systems down, but we have all the “right” testing documentation.]

30. “Total test automation will make testing much faster and more efficient, and we can save on expensive labor costs.” [We haven’t thought about who will design or script the automated tests, and we have no idea what it will take to maintain them.]


Lies told by customers

31. Exaggerating the risk of bugs to make a vendor or internal IT team look bad

32. Lying about their own state of preparedness

33. Falsely claiming a vendor’s solution was not technically viable [Customer’s IT had failed in two attempts to develop the system and didn’t want the vendor to succeed and show them up. I've seen variants of this elsewhere.]

Miscellaneous lies (anybody)

34. “We’re a month behind, but it won’t impact the schedule. We can make up the time!”

35. Knowingly committing to impossible deliveries

36. Deliberately downgrading bug severity to make a release

37. Falsely blaming slow progress on hold-ups by other teams or external vendors

38. Falsifying progress and status to look good

39. “Our security rules don’t permit you to have that access.” [Not true, but it’s a lot of work to give it to you, and I’m too busy/lazy.]