Quality Intelligence blog

Six Impossible Things Before Breakfast

2011-06-06T00:21:00.007-04:00

As an accompaniment to my June 2011 Tea-time with Testers article, "Six Impossible Things Before Breakfast", here is the list of lies I have heard on software projects over the years. I’ve grouped them roughly; arguably there are some lies listed that cross categories. Also, there is some overlap. “We have to get creative about these numbers!” is one variation on falsifying progress and status.

I invite you to add lies you've heard to my list. I'd love to hear about your experiences and observations of software project lying, and also of truth-telling in difficult project circumstances. Please leave a comment.

As a reminder, here are the parameters I set for my list.

I had personally to have heard the lie told one or more times. Each lie or category of lies had to be material to a software project, though it could have been told to make a sale before a project began or to describe a project after its end. Each lie had to be relatively common in the industry—or at least not rare. A lie had also to be significant to a project: to have influenced perceptions and/or decisions. And finally, I excluded malicious lies intended to subvert or sabotage an individual on a project.

Lies told to make a sale or get funding for a project

1. Deliberate underbidding [but we’ll make it up with change requests]

2. Bait & switch [proposing highly skilled staff to make the sale, then replacing them on the project with junior and lower-skilled people]

3. “The system will be standalone.” [To be usable, this particular system actually required expensive integration with several major systems, both upstream and downstream. The development team told management this before the project was approved, but the project sponsor ignored their information. I’ve heard similar claims on other projects.]

Management lies

4. Fictitious schedule: fixing scope, cost and staffing after arbitrarily slashing informed estimates by the project teams

5. “Our software is bug-free.”

6. “We can cut the testing in half without affecting quality.”

7. “This project is life or death for the company. If it doesn’t succeed, we’ll all be out of a job!”

8. “We’re slipping the date out, but don’t tell the team. We need to keep their feet to the fire.”

9. “We have to get creative about these numbers!”

10. “We can solve all the project’s problems with mandatory overtime.”

11. “The project team is pulling out all the stops to deliver this project.” [Actually, they’re exhausted and making a lot of mistakes, and we’re only getting about 30 hours productivity from the 60-hour weeks we’re making them work.]

Lies about programming

12. Code complete. [On one project, the programmers had actually left comments in the code listing what had not yet been done.]

13. “We’re done!” [Though we haven’t done the unit testing we committed to.]

14. “I only changed one line of code. You don’t need to test it.” [Though I didn’t do an impact analysis of the fix and I have no idea what might break.]

15. Padding estimates

16. Hiding bugs

17. “It’s not technically possible to do it that way.” [It is, actually, but I want to use this cool new technology that will enhance my skills.]

Lies about testing

18. “Anyone can test. We can get people off the street to do this job.”

19. “Anyone can test. We just have to give them the right process to follow.”

20. Tester exaggerating risk

21. Tester padding estimates

22. “This bug is out of my scope, and I’m under the gun. I don’t need to report it.”

23. “Testing is holding up the project. You’re finding too many bugs!”

24. “Our test cases will provide complete system coverage.”

25. “The infrastructure upgrade will be completely transparent. You won’t need to test.” [Although nobody did an impact analysis of the upgrade and we have no idea what might break.]

26. “We can deem these test cases passed.” [Though they haven’t actually been executed for months and we strongly suspect many of them wouldn’t pass.]

27. “You only need three weeks for testing.” [Because the code is late and we cut three weeks from the test schedule.]

28. “The testers don’t know what they’re doing.” [They estimated it would take two weeks, but found incomplete code and so many bugs it has taken six, and they’re not done yet.]

29. “Our mature test process employs all the industry best practices.” [Major bugs frequently bring production systems down, but we have all the “right” testing documentation.]

30. “Total test automation will make testing much faster and more efficient, and we can save on expensive labor costs.” [We haven’t thought about who will design or script the automated tests, and we have no idea what it will take to maintain them.]

Lies told by customers

31. Exaggerating the risk of bugs to make a vendor or internal IT team look bad

32. Lying about their own state of preparedness

33. Falsely claiming a vendor’s solution was not technically viable [Customer’s IT had failed in two attempts to develop the system and didn’t want the vendor to succeed and show them up. I've seen variants of this elsewhere.]

Miscellaneous lies (anybody)

34. “We’re a month behind, but it won’t impact the schedule. We can make up the time!”

35. Knowingly committing to impossible deliveries

36. Deliberately downgrading bug severity to make a release

37. Falsely blaming slow progress on hold-ups by other teams or external vendors

38. Falsifying progress and status to look good

39. “Our security rules don’t permit you to have that access.” [Not true, but it’s a lot of work to give it to you, and I’m too busy/lazy.]

Bug Severity vs. Priority

2010-08-22T10:53:00.001-04:00

Severity and priority are two ways of thinking about software bugs and deciding which ones should get fixed and in what order. Not everyone uses both. I’ve been in places where bugs are categorized by only one of them. But inevitably in those places, I find that the two ways of thinking get mixed and people end up confused. As I use them, bug severity and priority are different and we need both.

Severity is about the risk a bug poses if it gets out into the wild. I’ve written about testing and software risk elsewhere. See http://www.quality-intelligence.com/articles/DeterminingBusinessRisksForTesting.pdf.

Briefly, software risk can be characterized as:

The potential for
some fault, failure or other unintended happening
in the implemented system
to cause harm or loss
to one or more persons or organizations.

We assess the risk of a bug by asking questions about impact and probability. How much harm could this bug cause to some thing the customer cares about, such as human safety or the bottom line? How likely is this bug to manifest and how likely is that harm to occur if it does?

If a bug could cause significant harm but only manifests under very unlikely circumstances, then we might decide it’s less severe than a bug that could cause less harm but manifests frequently. Or not, depending on the context.

Most organizations have standard criteria for classifying bug severity, such as:

Severity 1 – Catastrophic bug or showstopper. Causes system crash, data corruption, irreparable harm, etc.

Severity 2 – Critical bug in important function. No reasonable workaround.

Severity 3 – Major bug but has viable workaround.

Severity 4 – Minor bug with trivial impact.

Typically, Severity 1 and 2 bugs must be fixed before release, where 3’s and 4’s might not be, depending on how many we have and on plans for their subsequent disposition.

Priority, on the other hand, is about the order in which bugs need to be fixed. Often, priority and severity run hand-in-hand: a bug is both high severity and high priority to fix. But that’s not always true. Occasionally in testing we’d like to have lower-severity bugs fixed so we can explore an area more thoroughly and see if they’re masking something else. Particularly on large projects, we can also find that we have a larger number of high severity bugs open than the programmers can readily fix. In this case, we need to specify the order for fixes based on where we plan to test next. To some extent also, ease of programming kicks in. A programmer working on high severity bugs in a particular module may choose to fix the low severity bugs in the same session.

At least theoretically, bug severity doesn’t change. The potential for business or technical impacts stays pretty much the same throughout the development project. (The passage of time can affect risk, but that’s a subject for another post.)

Priorities for fixing bugs do change depending on where we are in the project. At first, it’s testing priorities that matter most. But the closer we get to release, the more important the customer’s priorities become, to the point where they take over entirely.

And that brings us to an essential question about both severity and priority. Who gets to decide?

Ultimately, it’s the customer’s prerogative to decide both severity and priority (using “customer” as the stand-in here for “key decision-makers”). We—testers, project managers, and programmers alike—can make educated guesses about business risk and even about business tolerance for risk, but we can’t really know and we certainly can’t decide for the customer. Similarly, we can’t decide bug fix order for the customer, who frequently has different priorities from ours.

That’s not to say we abdicate responsibility. It’s a tester’s job to try and represent the customer’s point of view when they are absent. It’s also our job to help customers make those decisions. We do that by attempting to understand the true significance of a bug and communicating our understanding. We also ask questions to help our customers assess relative risk. (As anyone who has ever supported UAT knows, customers frequently assume everything is equally high risk until we ask those questions.)

It’s good to engage with customers and ask those questions early, ideally as we go through the development project. (Agile projects have it easy in this regard.) Early engagement is especially important for assigning severity.

But until we reach the point in the project of determining bug fix priorities for release, it’s only practical for the entire development team to set priorities according to what’s needed to move the project forward. Most often that means the testers’ needs: the bug fix priorities for testing.

Of course, there are other factors affecting priority during a development project, including the relative risk and cost of fixing a bug. But generally speaking, it makes sense for testers to drive priority until we switch over to the customer’s priorities for release.

Craftspeople testers

2010-06-04T06:22:00.001-04:00

An online forum I belong to is currently having a discussion about the “professional tester” and what that means.

As often happens to me, I find the question not quite framed to how I think. “Professional” is a word that has been fuzzed over time, from having a very precise application to wholesale acquisition by every coterie of white-collar workers who wanted more status—or, being charitable—to be taken seriously for their very real skills and (possible) contribution to society. Once, professionals were doctors and lawyers, and maybe engineers. Now, it’s apparently you, me, and Harry in the next cubicle who spends his days churning out code.

So I don’t find “professional” (tester or anything else) a useful label and I can’t be bothered wearing it. That’s as opposed to “professionalism”, which I think can perhaps still say something about a person’s conduct, ethics, and application of skill. “Craftsperson” is more interesting to me, as a concept, as a practice, and therefore as a possible handle.

My father was an engraver, a proud lifelong master practitioner of a highly skilled craft, and a constant explorer and learner of new skills. He was always practising, honing his skill. So I grew up with craft, and although I don’t remember my father ever articulating it this way, I learned the idea of craft as skill fuelled by love and integrity.

When I think about a “craftsperson tester” and what that means, I’m thinking about the career tester: the person who has chosen to stay with testing software for a living, however he or she got there. And I continually revise my definition of a good tester as I work on different projects and meet new people. I think there’s a great diversity in good testers that is too easily dismissed when we divide ourselves into “schools”, or even into communities of practice. I don’t do schools. The divisions—and divisiveness—practiced by some prominent testers (on both sides of the argument), bores me. I’ve said elsewhere that I’m not a card-carrying anything.

I’ve also said that I’m most in sympathy with the testers and thinkers in the Context Driven School. That remains true, in part because that’s where I see many craftsperson testers: people who, fuelled by love and integrity, continually strive to practice testing well, while growing themselves and the testing craft. And context is paramount to me.

But it’s not the only place I see craft. I see it also in what remains the mainstream: the big banks, the telecoms, and—yes—even among the ISEB-or-whatever-certified traditionalists who practice a pre-scripted form of testing they describe as “structured”. Although they’re often ignorant of other forms of testing, and uninterested in learning about them, many of these testers are highly skilled analysts and practitioners who are dedicated to testing software well. I’m working now with some excellent testers who practice testing during the working day and then go home and don’t think about it. Testing isn’t their life, and they don’t give a toss about the ferment of ideas and learning about testing that many of us constantly engage in.

But they’re good at what they do, dedicated to doing it well, and they hone their skills on the job. They teach and mentor others—on the job. I admire their thoughtfulness, skill, integrity and professionalism, and I certainly think of them as craftspeople. I enjoy working with them, and I count on them to do what’s needed on my project.

Of course, I know there are also bozos and seat-warmers among the traditionalists—large herds of them even, blighting the software and testing landscapes and giving us all a bad name (though not on my project!). But just because the good ones don’t fit my preferred model of craft, it’d be a big mistake to dismiss them.

I’m proud to work with craftspeople testers of all stripes.

With thanks to the members of Writing About Testing, whose discussion prompted this post.

Don't Argue with Sleepwalkers

2010-01-05T18:18:00.005-05:00

Influencing people is always hard. It’s especially difficult when people are operating from a fixed set of ideas with no room for even the possibility of a different point of view—almost as if they’re sleepwalking.

I thought of this recently when a friend said, “One thing I’ve learned over a long life: don’t argue with drunks.” “No”, I replied without missing a beat, “and don’t argue with sleepwalkers, either.” My friend didn’t get that one, so I explained.

In my youth, I was married to a guy who walked and talked in his sleep. Several times, I woke suddenly alone in bed with the kitchen light shining in my eyes. Shocked awake, I’d find him at the kitchen table eating the breakfast he’d just cooked—having showered, shaved and dressed fully including jacket and tie, all of it while fast asleep. If I hadn’t looked at the clock and I’d failed to notice his slightly glassy eyes, it would have seemed like a normal morning scene.

He talked coherently in his sleep, too. When I asked what he was doing, he’d say he was eating his breakfast and getting ready to go to work. At first, I was flabbergasted. I’d blurt out, “But it’s the middle of the night!“ And he’d say, “No, it’s eight thirty. Would you like some eggs?” Then I’d argue. But the more I tried to get him to see reality, the more adamant he became, almost angry. I’d be afraid that any minute now he’d go out the door, get in the car and drive there. And his unyielding obstinacy and escalating emotion disconcerted me.

I knew nothing about sleepwalking, not even the myth that waking a sleepwalker was dangerous. I think sometimes I did manage to wake him. Mostly though, the only way I could see out of the impasse was to play along, to try to enter my husband’s sleeping reality and while there try to persuade him to take a different course of action. I might say, “But Sweetie, it’s a holiday today and we can sleep in. You can go back to bed!” Then he’d cheerfully do that, and in the morning he’d have forgotten the entire episode.

One snowy night, I found him stark naked opening the front door: “waiting for the people to come”. By then I knew what to say. “Oh…but they aren’t coming for a couple of hours yet. So you might as well come back to bed and be warm in the meantime.”

The thing about sleepwalkers, I learned (and sleep-talkers, too), is that they can’t see outside the single frame of reference they’re stuck in. Argument is worse than futile. It upsets or annoys the sleeper. But if you can enter the sleeper’s world, you can operate within it to achieve a mutually acceptable result.

I try to keep this in mind whenever I have to deal with a manager or client whom I perceive as particularly blinkered or fixed on a single—and to me, wrong-headed—course of action. Perhaps it’s stretching a point to suggest such people are sleepwalking, but the situation is analogous enough that similar techniques can help.

When my ex-husband walked in his sleep, I had to:

Remind myself to approach him calmly and patiently, however absurd or alarming the situation he was in.
Listen to him and find out where he was coming from—his current frame of reference.
Refrain from arguing, but instead enter his frame of reference and suggest a different course of action consistent with it.

Compare that with this Aikido sequence that I learned from Jerry Weinberg: centre, enter, turn. That is:

Centre (yourself). Breathe calmly and concentrate your energy in the centre of your body.
Enter (your opponent’s attack).
Turn (your opponent’s attack to your own advantage or in the direction you want to go).

You can only ever influence people by connecting with their reality. That may not be so obvious when the other person’s reality has built-in points of connection with your own. But when there’s a radical disconnect between the other person’s frame of reference and your own, and he or she is taking a fixed position, it’s helpful to remember the sleepwalking analogy.

Next time a project manager gets totally stuck on the notion that you should publish metrics you’ve explained will be meaningless and misleading, don’t persist and tell her what a silly idea you think it is. Instead, act as if she’s talking in her sleep. Listen quietly and carefully to what she wants and where she’s coming from. Try and enter her frame of reference. Then go away and think about how you might operate within that frame of reference and turn what she’s demanding into something useful to both of you: say, a report that gives her a meaningful measurement you can believe in and support.

Don’t argue with sleepwalkers.