Breaking the Tyranny of Form - Part 1

Testing in many mainstream organizations is choked with low-value standardized documents that not only gobble up valuable thinking and testing time, but actively discourage thinking and impede good testing. While some testers can hope for relief from the document burden through the spread of Agile methods, this ridiculous situation isn't going away in a hurry. As a blog post by James Christie recently reminded us, the floodgates on “ISO/IEC 29119 Software Testing – the new international software testing standard” will soon open. I suppose it's possible that the new standard will wash away the documentation excesses we have now. I'm not holding my breath on that.

Test documents, whose sole purpose should be to serve the work, are instead driving and constraining the work. Absurdly, form is dictating substance. When, as in this case, the form is obese and bloated, it sucks up and squanders all the energy that ought to go into the real thing.

Some testers (many, I hope) refuse to be tyrannized by the supremacy of form. I want to reach into the mainstream where form dominates and help testers there to join us. I want to help them learn to think better and think for themselves. This blog post is a step in that ongoing effort.

Another step is the webinar I presented last week for EuroSTAR: "Unburdening Testing - Finding the Balance Point for Test Documentation". (The Q&A from that session are in blog form on the EuroSTAR site.) The webinar is an introduction to the interactive tutorial I will present November 6 at EuroSTAR 2012: "Right-sizing Test Documentation".

After presenting my own webinar, I watched the recorded version of Alan Richardson's excellent webinar "Thinking Visually in Software Testing". The thinking tools and practices he describes there are so uncannily like my own that it prompted me in writing this post to think and write about my thinking. (I encourage you to watch Alan's webinar, if you haven't already done so.) 

Form over Substance

When I first began working for a consulting company, my project manager gave me a mantra for billable work: “Never create anything that is not a deliverable to the customer”.

She was a brilliant PM who became an important mentor for me, but not all her advice was equally stellar. That statement in particular put horrible shackles on my work that I took years to shed completely.

The problem was that my customer deliverables were formal, standardized prose documents. For me to show value, I needed to end each day with sections and sub-sections populated with tidy paragraphs, building up to the finished product.

At first, I was lucky. My company was only dimly aware of standards purporting to govern test documentation, and so I created my own templates. But over time, my templates became our company standards. When I used them, I was so busy trying to adapt the structures for each project that I could no longer work easily with them. Like the templates for test documentation imposed in many companies, I found that the structure—the form—acted as a constraint on the substance. The tail was wagging the dog.

We Can't Test Without Thinking

Testing is all about thinking. We think and rethink constantly. We think about how best to gather information on our projects: what to read and look at, and who to talk to and when. We think about how to approach the software and we develop test ideas as we go. We create test models, often complementary models for testing different aspects of the same piece of software. We plan and replan and then plan again. We think about what we've discovered in testing and design what we're going to do next. If we're doing a good job, we don't ever stop thinking.

Prose documents in preset patterns inhibit thinking and creativity. It’s useful sometimes to have a checklist of important things to think about, but we can’t afford to let those checklists limit our thinking.Templates are not the best checklists.

Writing in sentences can sometimes help me to simplify and think through a tangled knot of ideas. I do occasionally write to understand what I’m thinking.  But I never set out to think in predetermined sections of formal standardized prose. Do you? Does anyone? Can anyone?

Visual Thinking

  A couple of years ago these drawings helped me think through a problem

Most often, I think in scribbles and doodles, beginning with notes and drawings that acquire structure as I develop the ideas and concepts. Or I might start with a tentative visual structure to generate ideas and then modify or replace the structure as needed to fit my thinking. Sometimes I scrawl ideas on coloured sticky notes and move them around over several days on a board or double-page spread of my notebook, drawing connections and annotating as I go. I often use mindmaps. I may use several different techniques to get my hands around a difficult problem. 

Diagrams Emerge

What comes out of my initial thinking processes is rarely a customer deliverable. But over time, the result is usually some kind of structured diagram or set of diagrams that I can then use to communicate my ideas to other people. Rather than dictating and constraining the substance, the form of these diagrams emerges from the substance.

Example of a strategy diagram for an end-to-end systems integration test on a large project

When—as is true on most projects—I must produce a formal document, I prefer to put diagrams front and centre. I want my documents to communicate, and I try to make them easy to read and understand. I use prose as sparingly as I can get away with, using tables and lists wherever possible.  I don't include boilerplate, and I never copy wodges of text from one document to another. (Why ever would I waste valuable project time on such useless make-work?)

This diagram shows the division of responsibility for testing on the same big project

Vacuous Form Tyrannizes the Mainstream

Apparently, that's not how most testers and test leads develop test documentation. In my consulting work with clients, I constantly see mammoth documents stuffed with books worth of low-content stodgy and opaque prose. Often, I search so-called test strategy documents in vain for any actual strategy. I fall asleep looking at test scripts that dictate every point and click and hideously repeat over and over again the most minute detail of so-called "test steps" and their piddling expected results. It's very hard to believe that much thinking is represented therein—or will inform the testing that must unfortunately follow.


Nobody reads this stuff. It isn't useful. It certainly doesn't help anyone test well. So why do testers waste time and spirit churning it out? Why do their managers insist on the tyranny of form over the substance that only thought can produce?

Let's Take Testing Back!

I believe that thinking testers must take testing back from the process weenies and form merchants.  Many testers have done this, but it has yet to happen in the mainstream—in big companies, big banks, government projects...sometimes even in startups and small companies. 

In subsequent posts on this topic, I'll explore some ways we can do this.

Let's Test 2012 - A Personal View

 Let's Test 2012

I'm rarely tempted to write about conferences, but for once I can't resist: the inaugural Let's Test was a wonderful experience in so many ways.

The Place

The rural setting at Runo, with its stunning "nobility of labour" art and bright, airy buildings, was an inspired choice, and no small contributor to the conference's success. We were all captive--miles from other (lesser) attractions--and there were many comfortable and inviting places to sit and confer, both indoors and out. It didn't hurt that there was a bar (with free drinks provided by sponsors on a couple of evenings), and fabulous food (the smoked salmon!). All meals were provided and the breaks were generously provisioned with treats like raspberry smoothies or pretty little macarons. When did you ever eat well, or even sit comfortably, at a conference?

Organizer Johan Jonasson, who knows I carry my own coffee to conferences and frequently joins me in a cup, promised me good Swedish coffee at Let's Test. It wasn't brilliant, but it was indeed decently strong and far more drinkable than I'd ever get in a North American hotel.

The Participants 

I use the word advisedly. The people at Let's Test were active participants throughout. Adopting the CAST facilitation model may have helped, and it certainly kept in-session discussions on track, but I don't think anything would have kept this group from lively and interesting conversations, often far into the night. And the Test Lab drew a big crowd every evening. That's what conferences are for!

The Sessions

People like me who present at a lot of conferences can become very jaded. At most, I rarely find the official sessions compelling and tend to skip them in favour of corridor conversations and networking. In contrast, the Let's Test agenda presented the best dilemma I can experience at a conference. In several time slots, I had difficulty deciding which session to go to. I was glad I chose Christin Wiedemann's "You Are a Scientist - Embracing the Scientific Method in Software Testing"  and Alan Richardson's "Testing Hypnotically", but sorry to miss several others. The real standout for me was Anne-Marie's Charrett's "Coaching Testers", especially the fascinating live session at the end, where many of us got to offer suggestions to (i.e., "coach") the coaches.

CAST is the only other test conference I know of that has this "assembly of peers" feeling, where speakers are as interested in other people's sessions as they are in their own.  

Presenting

I presented a full-day tutorial on Test Leadership and a track session on managing an end-to-end systems integration test.

I felt a little trepidation before the tutorial. How would a big group (28 people), for the most part meeting me and each other for the first time, and speaking what was for most a second language, work together in an experiential session? Would they embrace exercises that might seem strange to them, and then be willing to share their experiences and learnings in a debrief?

I needn't have worried. I started with a small-group exercise that probably helped to draw participants in, particularly some of the quieter ones. In that first exercise, one of the small groups--all Swedes--sensibly worked together in Swedish, though their output was in English. Otherwise (to my shame as a less than marginally bilingual North American), this multilingual group blew me away with their facility and willingness to communicate in my language. Soon everyone was engaged and working together, and they stayed engaged throughout the day (some more quietly than others), coming up with many fascinating insights and strategies for dealing with issues in test leadership.

I am grateful to everyone in my class for making it such a great learning experience for me! So far, I've had very positive feedback from participants. If you were there, please let me know what you think. Did the session work for you? What went well? What do you think I could do better next time?

Meanwhile, here's a link to Simon Morley's blog, where he describes his reactions.

Onwards (for me)!

I'll be keynoting (Are You Managing Test - or "The Test Process") at the BCS SIGIST in London on June 21, and also doing a workshop on mind-mapping a test strategy. I look forward to working with a group of engaged British testers there.

June 26-28, I will be presenting Beyond Process: three 1-day experiential classes in London, including a reprisal of my Let's Test tutorial, "Inspiring Testing" plus "Determining Business Risks for Testing" and "The Design behind the Plan - Test Strategy workshop". Registration is through the sponsor, ElectroMind.

If you're in North America, you may want to register for Beyond Process in San Jose before CAST, July 11-13.

Then it will be time for CAST 2012, July 16-18. As Program Chair, I believe the Program Committee and I have assembled an excellent program of sessions and innovative workshops. I'm really looking forward to seeing some of my new and old friends from Let's Test, as well as many others who didn't make it to Sweden. It too is going to be a wonderful conference!

Onwards (for Let's Test)!

Let's Test 2012 was a terrific first conference. My thanks and congratulations to the organizers, and I look forward eagerly to Let's Test 2013!

And some pics from the Nature Reserve

One of the evening activities was a Nature Walk, guided by a marvelously fit 80-year-old gentleman,  recently named Sweden's Gardener of the Year. It was so beautiful, I rose early the next morning and went out again with my camera.

Six Impossible Things Before Breakfast

As an accompaniment to my June 2011 Tea-time with Testers article, "Six Impossible Things Before Breakfast", here is the list of lies I have heard on software projects over the years. I’ve grouped them roughly; arguably there are some lies listed that cross categories. Also, there is some overlap. “We have to get creative about these numbers!” is one variation on falsifying progress and status.

I invite you to add lies you've heard to my list. I'd love to hear about your experiences and observations of software project lying, and also of truth-telling in difficult project circumstances. Please leave a comment.

As a reminder, here are the parameters I set for my list.

I had personally to have heard the lie told one or more times. Each lie or category of lies had to be material to a software project, though it could have been told to make a sale before a project began or to describe a project after its end. Each lie had to be relatively common in the industry—or at least not rare. A lie had also to be significant to a project: to have influenced perceptions and/or decisions. And finally, I excluded malicious lies intended to subvert or sabotage an individual on a project.

Lies told to make a sale or get funding for a project

1. Deliberate underbidding [but we’ll make it up with change requests]

2. Bait & switch [proposing highly skilled staff to make the sale, then replacing them on the project with junior and lower-skilled people]

3. “The system will be standalone.” [To be usable, this particular system actually required expensive integration with several major systems, both upstream and downstream. The development team told management this before the project was approved, but the project sponsor ignored their information. I’ve heard similar claims on other projects.]

Management lies

4. Fictitious schedule: fixing scope, cost and staffing after arbitrarily slashing informed estimates by the project teams

5. “Our software is bug-free.”

6. “We can cut the testing in half without affecting quality.”

7. “This project is life or death for the company. If it doesn’t succeed, we’ll all be out of a job!”

8. “We’re slipping the date out, but don’t tell the team. We need to keep their feet to the fire.”

9. “We have to get creative about these numbers!”

10. “We can solve all the project’s problems with mandatory overtime.”

11. “The project team is pulling out all the stops to deliver this project.” [Actually, they’re exhausted and making a lot of mistakes, and we’re only getting about 30 hours productivity from the 60-hour weeks we’re making them work.]

Lies about programming

12. Code complete. [On one project, the programmers had actually left comments in the code listing what had not yet been done.]

13. “We’re done!” [Though we haven’t done the unit testing we committed to.]

14. “I only changed one line of code. You don’t need to test it.” [Though I didn’t do an impact analysis of the fix and I have no idea what might break.]

15. Padding estimates

16. Hiding bugs

17. “It’s not technically possible to do it that way.” [It is, actually, but I want to use this cool new technology that will enhance my skills.]

Lies about testing

18. “Anyone can test. We can get people off the street to do this job.”

19. “Anyone can test. We just have to give them the right process to follow.”

20. Tester exaggerating risk

21. Tester padding estimates

22. “This bug is out of my scope, and I’m under the gun. I don’t need to report it.”

23. “Testing is holding up the project. You’re finding too many bugs!”

24. “Our test cases will provide complete system coverage.”

25. “The infrastructure upgrade will be completely transparent. You won’t need to test.” [Although nobody did an impact analysis of the upgrade and we have no idea what might break.]

26. “We can deem these test cases passed.” [Though they haven’t actually been executed for months and we strongly suspect many of them wouldn’t pass.]

27. “You only need three weeks for testing.” [Because the code is late and we cut three weeks from the test schedule.]

28. “The testers don’t know what they’re doing.” [They estimated it would take two weeks, but found incomplete code and so many bugs it has taken six, and they’re not done yet.]

29. “Our mature test process employs all the industry best practices.” [Major bugs frequently bring production systems down, but we have all the “right” testing documentation.]

30. “Total test automation will make testing much faster and more efficient, and we can save on expensive labor costs.” [We haven’t thought about who will design or script the automated tests, and we have no idea what it will take to maintain them.]


Lies told by customers

31. Exaggerating the risk of bugs to make a vendor or internal IT team look bad

32. Lying about their own state of preparedness

33. Falsely claiming a vendor’s solution was not technically viable [Customer’s IT had failed in two attempts to develop the system and didn’t want the vendor to succeed and show them up. I've seen variants of this elsewhere.]

Miscellaneous lies (anybody)

34. “We’re a month behind, but it won’t impact the schedule. We can make up the time!”

35. Knowingly committing to impossible deliveries

36. Deliberately downgrading bug severity to make a release

37. Falsely blaming slow progress on hold-ups by other teams or external vendors

38. Falsifying progress and status to look good

39. “Our security rules don’t permit you to have that access.” [Not true, but it’s a lot of work to give it to you, and I’m too busy/lazy.]

Bug Severity vs. Priority

Severity and priority are two ways of thinking about software bugs and deciding which ones should get fixed and in what order. Not everyone uses both. I’ve been in places where bugs are categorized by only one of them. But inevitably in those places, I find that the two ways of thinking get mixed and people end up confused. As I use them, bug severity and priority are different and we need both.

Severity is about the risk a bug poses if it gets out into the wild. I’ve written about testing and software risk elsewhere. See http://www.quality-intelligence.com/articles/DeterminingBusinessRisksForTesting.pdf.

Briefly, software risk can be characterized as:

The potential for
some fault, failure or other unintended happening
in the implemented system
to cause harm or loss
to one or more persons or organizations.

We assess the risk of a bug by asking questions about impact and probability. How much harm could this bug cause to some thing the customer cares about, such as human safety or the bottom line? How likely is this bug to manifest and how likely is that harm to occur if it does?

If a bug could cause significant harm but only manifests under very unlikely circumstances, then we might decide it’s less severe than a bug that could cause less harm but manifests frequently. Or not, depending on the context.

Most organizations have standard criteria for classifying bug severity, such as:

Severity 1 – Catastrophic bug or showstopper. Causes system crash, data corruption, irreparable harm, etc.

Severity 2 – Critical bug in important function. No reasonable workaround.

Severity 3 – Major bug but has viable workaround.

Severity 4 – Minor bug with trivial impact.

Typically, Severity 1 and 2 bugs must be fixed before release, where 3’s and 4’s might not be, depending on how many we have and on plans for their subsequent disposition.

Priority, on the other hand, is about the order in which bugs need to be fixed. Often, priority and severity run hand-in-hand: a bug is both high severity and high priority to fix. But that’s not always true. Occasionally in testing we’d like to have lower-severity bugs fixed so we can explore an area more thoroughly and see if they’re masking something else. Particularly on large projects, we can also find that we have a larger number of high severity bugs open than the programmers can readily fix. In this case, we need to specify the order for fixes based on where we plan to test next. To some extent also, ease of programming kicks in. A programmer working on high severity bugs in a particular module may choose to fix the low severity bugs in the same session.

At least theoretically, bug severity doesn’t change. The potential for business or technical impacts stays pretty much the same throughout the development project. (The passage of time can affect risk, but that’s a subject for another post.)

Priorities for fixing bugs do change depending on where we are in the project. At first, it’s testing priorities that matter most. But the closer we get to release, the more important the customer’s priorities become, to the point where they take over entirely.

And that brings us to an essential question about both severity and priority. Who gets to decide?

Ultimately, it’s the customer’s prerogative to decide both severity and priority (using “customer” as the stand-in here for “key decision-makers”). We—testers, project managers, and programmers alike—can make educated guesses about business risk and even about business tolerance for risk, but we can’t really know and we certainly can’t decide for the customer. Similarly, we can’t decide bug fix order for the customer, who frequently has different priorities from ours.

That’s not to say we abdicate responsibility. It’s a tester’s job to try and represent the customer’s point of view when they are absent. It’s also our job to help customers make those decisions. We do that by attempting to understand the true significance of a bug and communicating our understanding. We also ask questions to help our customers assess relative risk. (As anyone who has ever supported UAT knows, customers frequently assume everything is equally high risk until we ask those questions.)

It’s good to engage with customers and ask those questions early, ideally as we go through the development project. (Agile projects have it easy in this regard.) Early engagement is especially important for assigning severity.

But until we reach the point in the project of determining bug fix priorities for release, it’s only practical for the entire development team to set priorities according to what’s needed to move the project forward. Most often that means the testers’ needs: the bug fix priorities for testing.

Of course, there are other factors affecting priority during a development project, including the relative risk and cost of fixing a bug. But generally speaking, it makes sense for testers to drive priority until we switch over to the customer’s priorities for release.

Craftspeople testers

An online forum I belong to is currently having a discussion about the “professional tester” and what that means.

As often happens to me, I find the question not quite framed to how I think. “Professional” is a word that has been fuzzed over time, from having a very precise application to wholesale acquisition by every coterie of white-collar workers who wanted more status—or, being charitable—to be taken seriously for their very real skills and (possible) contribution to society. Once, professionals were doctors and lawyers, and maybe engineers. Now, it’s apparently you, me, and Harry in the next cubicle who spends his days churning out code.

So I don’t find “professional” (tester or anything else) a useful label and I can’t be bothered wearing it. That’s as opposed to “professionalism”, which I think can perhaps still say something about a person’s conduct, ethics, and application of skill. “Craftsperson” is more interesting to me, as a concept, as a practice, and therefore as a possible handle.

My father was an engraver, a proud lifelong master practitioner of a highly skilled craft, and a constant explorer and learner of new skills. He was always practising, honing his skill. So I grew up with craft, and although I don’t remember my father ever articulating it this way, I learned the idea of craft as skill fuelled by love and integrity.

When I think about a “craftsperson tester” and what that means, I’m thinking about the career tester: the person who has chosen to stay with testing software for a living, however he or she got there. And I continually revise my definition of a good tester as I work on different projects and meet new people. I think there’s a great diversity in good testers that is too easily dismissed when we divide ourselves into “schools”, or even into communities of practice. I don’t do schools. The divisions—and divisiveness—practiced by some prominent testers (on both sides of the argument), bores me. I’ve said elsewhere that I’m not a card-carrying anything.

I’ve also said that I’m most in sympathy with the testers and thinkers in the Context Driven School. That remains true, in part because that’s where I see many craftsperson testers: people who, fuelled by love and integrity, continually strive to practice testing well, while growing themselves and the testing craft. And context is paramount to me.

But it’s not the only place I see craft. I see it also in what remains the mainstream: the big banks, the telecoms, and—yes—even among the ISEB-or-whatever-certified traditionalists who practice a pre-scripted form of testing they describe as “structured”. Although they’re often ignorant of other forms of testing, and uninterested in learning about them, many of these testers are highly skilled analysts and practitioners who are dedicated to testing software well. I’m working now with some excellent testers who practice testing during the working day and then go home and don’t think about it. Testing isn’t their life, and they don’t give a toss about the ferment of ideas and learning about testing that many of us constantly engage in.

But they’re good at what they do, dedicated to doing it well, and they hone their skills on the job. They teach and mentor others—on the job. I admire their thoughtfulness, skill, integrity and professionalism, and I certainly think of them as craftspeople. I enjoy working with them, and I count on them to do what’s needed on my project.

Of course, I know there are also bozos and seat-warmers among the traditionalists—large herds of them even, blighting the software and testing landscapes and giving us all a bad name (though not on my project!). But just because the good ones don’t fit my preferred model of craft, it’d be a big mistake to dismiss them.

I’m proud to work with craftspeople testers of all stripes.

With thanks to the members of Writing About Testing, whose discussion prompted this post.

Don't Argue with Sleepwalkers

Influencing people is always hard. It’s especially difficult when people are operating from a fixed set of ideas with no room for even the possibility of a different point of view—almost as if they’re sleepwalking.

I thought of this recently when a friend said, “One thing I’ve learned over a long life: don’t argue with drunks.” “No”, I replied without missing a beat, “and don’t argue with sleepwalkers, either.” My friend didn’t get that one, so I explained.

In my youth, I was married to a guy who walked and talked in his sleep. Several times, I woke suddenly alone in bed with the kitchen light shining in my eyes. Shocked awake, I’d find him at the kitchen table eating the breakfast he’d just cooked—having showered, shaved and dressed fully including jacket and tie, all of it while fast asleep. If I hadn’t looked at the clock and I’d failed to notice his slightly glassy eyes, it would have seemed like a normal morning scene.

He talked coherently in his sleep, too. When I asked what he was doing, he’d say he was eating his breakfast and getting ready to go to work. At first, I was flabbergasted. I’d blurt out, “But it’s the middle of the night!“ And he’d say, “No, it’s eight thirty. Would you like some eggs?” Then I’d argue. But the more I tried to get him to see reality, the more adamant he became, almost angry. I’d be afraid that any minute now he’d go out the door, get in the car and drive there. And his unyielding obstinacy and escalating emotion disconcerted me.

I knew nothing about sleepwalking, not even the myth that waking a sleepwalker was dangerous. I think sometimes I did manage to wake him. Mostly though, the only way I could see out of the impasse was to play along, to try to enter my husband’s sleeping reality and while there try to persuade him to take a different course of action. I might say, “But Sweetie, it’s a holiday today and we can sleep in. You can go back to bed!” Then he’d cheerfully do that, and in the morning he’d have forgotten the entire episode.

One snowy night, I found him stark naked opening the front door: “waiting for the people to come”. By then I knew what to say. “Oh…but they aren’t coming for a couple of hours yet. So you might as well come back to bed and be warm in the meantime.”

The thing about sleepwalkers, I learned (and sleep-talkers, too), is that they can’t see outside the single frame of reference they’re stuck in. Argument is worse than futile. It upsets or annoys the sleeper. But if you can enter the sleeper’s world, you can operate within it to achieve a mutually acceptable result.

I try to keep this in mind whenever I have to deal with a manager or client whom I perceive as particularly blinkered or fixed on a single—and to me, wrong-headed—course of action. Perhaps it’s stretching a point to suggest such people are sleepwalking, but the situation is analogous enough that similar techniques can help.

When my ex-husband walked in his sleep, I had to:

• Remind myself to approach him calmly and patiently, however absurd or alarming the situation he was in.
  
  
• Listen to him and find out where he was coming from—his current frame of reference.
  
  
• Refrain from arguing, but instead enter his frame of reference and suggest a different course of action consistent with it.
  
  

Compare that with this Aikido sequence that I learned from Jerry Weinberg: centre, enter, turn. That is:

• Centre (yourself). Breathe calmly and concentrate your energy in the centre of your body.
  
  
• Enter (your opponent’s attack).
  
  
• Turn (your opponent’s attack to your own advantage or in the direction you want to go).
  
  

You can only ever influence people by connecting with their reality. That may not be so obvious when the other person’s reality has built-in points of connection with your own. But when there’s a radical disconnect between the other person’s frame of reference and your own, and he or she is taking a fixed position, it’s helpful to remember the sleepwalking analogy.

Next time a project manager gets totally stuck on the notion that you should publish metrics you’ve explained will be meaningless and misleading, don’t persist and tell her what a silly idea you think it is. Instead, act as if she’s talking in her sleep. Listen quietly and carefully to what she wants and where she’s coming from. Try and enter her frame of reference. Then go away and think about how you might operate within that frame of reference and turn what she’s demanding into something useful to both of you: say, a report that gives her a meaningful measurement you can believe in and support.

Don’t argue with sleepwalkers.