I Changed My View On Estimating Bugs and So Can You

"You’ll have to know what to do. If you don’t know what to do you have to find out. If you can’t find out you bloody well make it up and then you make it so."

~ China Miéville, Kraken

I used to recommend that teams not estimate bugs. I was always uneasy about this topic, and recently I changed my mind. Here's why.

The Arguments Against Estimating Bugs

There are many arguments against estimating bugs, and here are the most prominent. I've used some of them myself to convince teams to drop the idea.

Bugs and Features Are Just Different

The idea here is that, unlike features, bugs are inherently unpredictable. By definition, a bug is a system behaving in unintended – and therefore unknown – ways, making it unfathomable.

While this sounds profound, it's simply not true. There's nothing mystical about a button being the wrong colour. It's not some strange behaviour of the system that only appears under rare and obscure circumstances. It just has the wrong colour. Of course you can estimate how much work it will take to fix that.

Bugs and Features Have Different Workflows

Typically, bugs enter the workflow differently from features. Features go into a backlog, get refined by the whole team—often through several iterations—and are eventually implemented. Bugs, however, are often handled by a subset of the team. The report is reviewed, the bug is reproduced and validated, severity is assessed, and a fix is implemented (or not).

This is substantially different from the feature workflow. And it seems impractical to use the same refinement-based workflow for bugs.

And while that's true, and it probably makes little sense to bring a bug report in front of the whole team, I fail to see why this is a good argument against estimating. Yes, you'd probably have to deviate from the "the whole team estimates everything" mantra to "a part of the team estimates this bug", but since you're implementing a different workflow for bugs anyway, this workflow could include estimates, too, couldn't it?

With Bugs, the Investigation Often Is the Fix

Bugs that slip through into production often fall into the category "Very difficult to understand, very easy to fix." Easily spotted defects that make it to production signal a breakdown in the development workflow; And it is not often that something that is so conceptually flawed that it is very hard to fix makes it into production. Yes, that happens, but if it happens on a regular basis, how to handle bugs isn't your top priority – your entire approach to development is.

So, once you understand a bug, fixing it can become trivial.

While there is a lot of truth to this, I fail to see how this is fundamentally different from implementing a new feature. Discovery is part of both.

Estimating Bugs Rewards the Team for Being Sloppy

If a team estimates bugs, fixing them counts towards velocity, suggesting that the team is more productive than it really is. So, creating bugs, then estimating and fixing them would create story points that are equivalent to the team implementing new functionality.

While this is technically true, and story points on bugs would automatically increase the velocity of the team, this argument reveals a poor understanding of the strengths and limitations of velocity.

Velocity is not a suitable metric for success, or progress. It is not even an acceptable proxy, or a useful substitute.

Velocity tells you something about the capacity of the flow system, but nothing about the usefulness of what the flow system creates. In that regard, estimating bugs makes perfect sense: Bug fixes are part of that capacity.

If you’re worried that estimating bugs makes your team look more successful than they are, you probably need to reconsider how you measure success in the first place.

Estimating Bugs Rewards the Team Being Sloppy, Part 2

A variation of the previous point goes like this:
If a team estimates bugs, there’s no incentive to fix bugs discovered during in-sprint work. In fact, there’s a perverse incentive to not fix them right away. Why? Because fixing a bug on the spot adds effort, but doesn’t contribute to velocity. By contrast, if the team files a bug, they can move the feature to “done” (even though it’s buggy), estimate the bug separately, and earn story points for fixing it later.

I am not making this up. I have heard a Scrum Master defending their stance of not estimating bugs by using this argument.

The only thing I have to say is: If this is really the state of affairs in your team, and you suspect the team would start to behave in this way, there is something more fundamentally wrong than any discussion about the finer points of estimations could ever fix.

The Arguments For Estimating Bugs

Now let’s look at why estimating bugs is not only valid, but theoretically sound (as far as anything in agile estimation is theoretically sound) and potentially useful.

Work Is Work

This is the obvious one. Work is work, and when working on bugs, you will not be working on something else. How much work those bugs probably are going to be is useful information both for planning, but also for understanding overall workload. Yes, you could reserve a fixed amount of time of the team to work on bugs, but why use a different method for sizing work depending on wether you're creating something new, or fixing something broken?

It Helps with Sprint Planning

Can the people assigned to the bug handle it on their own, or do they need support from the rest of the team? How does that impact the overall sprint?

If you estimate the bug, you can use your existing mechanisms for capacity planning and prioritisation. That’s useful.

Sizing Bugs Gives You a Valuable Metric

In an estimation-less environment, all bugs are created equal. You will assign a severity to them (typically a mix of impact and reach), but the effort needed to fix them is invisible.

Estimation adds that dimension to your data. It allows you to better understand the quality of your product, and help with trade-off decisions.

Maybe there is a bug, but it is exceptionally hard to fix, and there is a well-established workaround? Should we fix that bug now? Or should we fix it later?

I know, many people will drop their cup of coffee upon reading such heresy, but these sorts of decisions are the reality of many teams. It makes no sense to pretend these problems aren't there.

Bug Size Supports More Realistic SLAs

Yes, I know. Customers don't care about our problems. They just want things fixed. SLAs should be based on defect severity, not our own struggles fixing them.

But resources are limited. Having size-based estimates can help set more realistic expectations and timelines. That’s just being honest.

Why I Think The Arguments For Estimating Bugs Outweigh The Arguments Against

To put one thing out of the way first: If you'd rather not estimate bugs, that's totally fine. I wouldn't say that you have to. I wouldn't even say that you should. But overall, I do think the benefits outweigh the downsides.

Using the same unit system for sizing features and bugs (e.g., Story Points) allows you to apply all the stuff you have for reporting, planning and decision-making to bugs as well - and that's quite useful.

You don't have to do the mental mechanics of "we use 10% of our time for bug fixes," because that opens a can of worms, each worm a nasty follow-up question. Do you block work, or is it continuous effort? Do the 10% apply even if there are no serious bugs, but we're behind on a feature? What if there's a bug that can't be fixed in 10% of our time?

Practical Aspects of Estimating Bugs

Small, Medium, Large

Fixing something is different from creating something new, no doubt about that. As I've said, more often than not, understanding the issue will take way more time than fixing it. In that sense, understanding the work more or less is the work.

And since understanding the work is the basis for estimating it, we are in danger of having everything backwards - understanding where the bug comes from, then stopping working on it, estimating it, and then doing the fix.

That doesn't make a whole lot of sense. Instead, I recommend being much coarser and faster with bug estimation. Maybe use t-shirt sizes and then use a representation of "small", "medium" and "large" in your team's use of story points.

Don’t File Bugs Just to Shelf Them

Never let the practice of estimation be a gateway to a sloppy handling with your Definition of Done. Done means done, and that typically means "free from known defects". Don't move something that is not bug-free to Done and create a couple of bug tickets for it in the backlog.

We're in the Business of Fixing Bugs, Not Creating a Perfect System For Stacking the Reports

With apologies to Merlin Mann,¹ our job is to identify, understand, and fix bugs. Not creating the perfect system for managing the reports.

Whatever helps you in doing that is great. Everything else is just a distraction.