Bug Severity vs. Priority

Severity and priority are two ways of thinking about software bugs and deciding which ones should get fixed and in what order. Not everyone uses both. I’ve been in places where bugs are categorized by only one of them. But inevitably in those places, I find that the two ways of thinking get mixed and people end up confused. As I use them, bug severity and priority are different and we need both.

Severity is about the risk a bug poses if it gets out into the wild. I’ve written about testing and software risk elsewhere. See http://www.quality-intelligence.com/articles/DeterminingBusinessRisksForTesting.pdf.

Briefly, software risk can be characterized as:

The potential for
some fault, failure or other unintended happening
in the implemented system
to cause harm or loss
to one or more persons or organizations.

We assess the risk of a bug by asking questions about impact and probability. How much harm could this bug cause to some thing the customer cares about, such as human safety or the bottom line? How likely is this bug to manifest and how likely is that harm to occur if it does?

If a bug could cause significant harm but only manifests under very unlikely circumstances, then we might decide it’s less severe than a bug that could cause less harm but manifests frequently. Or not, depending on the context.

Most organizations have standard criteria for classifying bug severity, such as:

Severity 1 – Catastrophic bug or showstopper. Causes system crash, data corruption, irreparable harm, etc.

Severity 2 – Critical bug in important function. No reasonable workaround.

Severity 3 – Major bug but has viable workaround.

Severity 4 – Minor bug with trivial impact.

Typically, Severity 1 and 2 bugs must be fixed before release, where 3’s and 4’s might not be, depending on how many we have and on plans for their subsequent disposition.

Priority, on the other hand, is about the order in which bugs need to be fixed. Often, priority and severity run hand-in-hand: a bug is both high severity and high priority to fix. But that’s not always true. Occasionally in testing we’d like to have lower-severity bugs fixed so we can explore an area more thoroughly and see if they’re masking something else. Particularly on large projects, we can also find that we have a larger number of high severity bugs open than the programmers can readily fix. In this case, we need to specify the order for fixes based on where we plan to test next. To some extent also, ease of programming kicks in. A programmer working on high severity bugs in a particular module may choose to fix the low severity bugs in the same session.

At least theoretically, bug severity doesn’t change. The potential for business or technical impacts stays pretty much the same throughout the development project. (The passage of time can affect risk, but that’s a subject for another post.)

Priorities for fixing bugs do change depending on where we are in the project. At first, it’s testing priorities that matter most. But the closer we get to release, the more important the customer’s priorities become, to the point where they take over entirely.

And that brings us to an essential question about both severity and priority. Who gets to decide?

Ultimately, it’s the customer’s prerogative to decide both severity and priority (using “customer” as the stand-in here for “key decision-makers”). We—testers, project managers, and programmers alike—can make educated guesses about business risk and even about business tolerance for risk, but we can’t really know and we certainly can’t decide for the customer. Similarly, we can’t decide bug fix order for the customer, who frequently has different priorities from ours.

That’s not to say we abdicate responsibility. It’s a tester’s job to try and represent the customer’s point of view when they are absent. It’s also our job to help customers make those decisions. We do that by attempting to understand the true significance of a bug and communicating our understanding. We also ask questions to help our customers assess relative risk. (As anyone who has ever supported UAT knows, customers frequently assume everything is equally high risk until we ask those questions.)

It’s good to engage with customers and ask those questions early, ideally as we go through the development project. (Agile projects have it easy in this regard.) Early engagement is especially important for assigning severity.

But until we reach the point in the project of determining bug fix priorities for release, it’s only practical for the entire development team to set priorities according to what’s needed to move the project forward. Most often that means the testers’ needs: the bug fix priorities for testing.

Of course, there are other factors affecting priority during a development project, including the relative risk and cost of fixing a bug. But generally speaking, it makes sense for testers to drive priority until we switch over to the customer’s priorities for release.

Craftspeople testers

An online forum I belong to is currently having a discussion about the “professional tester” and what that means.

As often happens to me, I find the question not quite framed to how I think. “Professional” is a word that has been fuzzed over time, from having a very precise application to wholesale acquisition by every coterie of white-collar workers who wanted more status—or, being charitable—to be taken seriously for their very real skills and (possible) contribution to society. Once, professionals were doctors and lawyers, and maybe engineers. Now, it’s apparently you, me, and Harry in the next cubicle who spends his days churning out code.

So I don’t find “professional” (tester or anything else) a useful label and I can’t be bothered wearing it. That’s as opposed to “professionalism”, which I think can perhaps still say something about a person’s conduct, ethics, and application of skill. “Craftsperson” is more interesting to me, as a concept, as a practice, and therefore as a possible handle.

My father was an engraver, a proud lifelong master practitioner of a highly skilled craft, and a constant explorer and learner of new skills. He was always practising, honing his skill. So I grew up with craft, and although I don’t remember my father ever articulating it this way, I learned the idea of craft as skill fuelled by love and integrity.

When I think about a “craftsperson tester” and what that means, I’m thinking about the career tester: the person who has chosen to stay with testing software for a living, however he or she got there. And I continually revise my definition of a good tester as I work on different projects and meet new people. I think there’s a great diversity in good testers that is too easily dismissed when we divide ourselves into “schools”, or even into communities of practice. I don’t do schools. The divisions—and divisiveness—practiced by some prominent testers (on both sides of the argument), bores me. I’ve said elsewhere that I’m not a card-carrying anything.

I’ve also said that I’m most in sympathy with the testers and thinkers in the Context Driven School. That remains true, in part because that’s where I see many craftsperson testers: people who, fuelled by love and integrity, continually strive to practice testing well, while growing themselves and the testing craft. And context is paramount to me.

But it’s not the only place I see craft. I see it also in what remains the mainstream: the big banks, the telecoms, and—yes—even among the ISEB-or-whatever-certified traditionalists who practice a pre-scripted form of testing they describe as “structured”. Although they’re often ignorant of other forms of testing, and uninterested in learning about them, many of these testers are highly skilled analysts and practitioners who are dedicated to testing software well. I’m working now with some excellent testers who practice testing during the working day and then go home and don’t think about it. Testing isn’t their life, and they don’t give a toss about the ferment of ideas and learning about testing that many of us constantly engage in.

But they’re good at what they do, dedicated to doing it well, and they hone their skills on the job. They teach and mentor others—on the job. I admire their thoughtfulness, skill, integrity and professionalism, and I certainly think of them as craftspeople. I enjoy working with them, and I count on them to do what’s needed on my project.

Of course, I know there are also bozos and seat-warmers among the traditionalists—large herds of them even, blighting the software and testing landscapes and giving us all a bad name (though not on my project!). But just because the good ones don’t fit my preferred model of craft, it’d be a big mistake to dismiss them.

I’m proud to work with craftspeople testers of all stripes.

With thanks to the members of Writing About Testing, whose discussion prompted this post.

Don't Argue with Sleepwalkers

Influencing people is always hard. It’s especially difficult when people are operating from a fixed set of ideas with no room for even the possibility of a different point of view—almost as if they’re sleepwalking.

I thought of this recently when a friend said, “One thing I’ve learned over a long life: don’t argue with drunks.” “No”, I replied without missing a beat, “and don’t argue with sleepwalkers, either.” My friend didn’t get that one, so I explained.

In my youth, I was married to a guy who walked and talked in his sleep. Several times, I woke suddenly alone in bed with the kitchen light shining in my eyes. Shocked awake, I’d find him at the kitchen table eating the breakfast he’d just cooked—having showered, shaved and dressed fully including jacket and tie, all of it while fast asleep. If I hadn’t looked at the clock and I’d failed to notice his slightly glassy eyes, it would have seemed like a normal morning scene.

He talked coherently in his sleep, too. When I asked what he was doing, he’d say he was eating his breakfast and getting ready to go to work. At first, I was flabbergasted. I’d blurt out, “But it’s the middle of the night!“ And he’d say, “No, it’s eight thirty. Would you like some eggs?” Then I’d argue. But the more I tried to get him to see reality, the more adamant he became, almost angry. I’d be afraid that any minute now he’d go out the door, get in the car and drive there. And his unyielding obstinacy and escalating emotion disconcerted me.

I knew nothing about sleepwalking, not even the myth that waking a sleepwalker was dangerous. I think sometimes I did manage to wake him. Mostly though, the only way I could see out of the impasse was to play along, to try to enter my husband’s sleeping reality and while there try to persuade him to take a different course of action. I might say, “But Sweetie, it’s a holiday today and we can sleep in. You can go back to bed!” Then he’d cheerfully do that, and in the morning he’d have forgotten the entire episode.

One snowy night, I found him stark naked opening the front door: “waiting for the people to come”. By then I knew what to say. “Oh…but they aren’t coming for a couple of hours yet. So you might as well come back to bed and be warm in the meantime.”

The thing about sleepwalkers, I learned (and sleep-talkers, too), is that they can’t see outside the single frame of reference they’re stuck in. Argument is worse than futile. It upsets or annoys the sleeper. But if you can enter the sleeper’s world, you can operate within it to achieve a mutually acceptable result.

I try to keep this in mind whenever I have to deal with a manager or client whom I perceive as particularly blinkered or fixed on a single—and to me, wrong-headed—course of action. Perhaps it’s stretching a point to suggest such people are sleepwalking, but the situation is analogous enough that similar techniques can help.

When my ex-husband walked in his sleep, I had to:

• Remind myself to approach him calmly and patiently, however absurd or alarming the situation he was in.
  
  
• Listen to him and find out where he was coming from—his current frame of reference.
  
  
• Refrain from arguing, but instead enter his frame of reference and suggest a different course of action consistent with it.
  
  

Compare that with this Aikido sequence that I learned from Jerry Weinberg: centre, enter, turn. That is:

• Centre (yourself). Breathe calmly and concentrate your energy in the centre of your body.
  
  
• Enter (your opponent’s attack).
  
  
• Turn (your opponent’s attack to your own advantage or in the direction you want to go).
  
  

You can only ever influence people by connecting with their reality. That may not be so obvious when the other person’s reality has built-in points of connection with your own. But when there’s a radical disconnect between the other person’s frame of reference and your own, and he or she is taking a fixed position, it’s helpful to remember the sleepwalking analogy.

Next time a project manager gets totally stuck on the notion that you should publish metrics you’ve explained will be meaningless and misleading, don’t persist and tell her what a silly idea you think it is. Instead, act as if she’s talking in her sleep. Listen quietly and carefully to what she wants and where she’s coming from. Try and enter her frame of reference. Then go away and think about how you might operate within that frame of reference and turn what she’s demanding into something useful to both of you: say, a report that gives her a meaningful measurement you can believe in and support.

Don’t argue with sleepwalkers.