Base rates, the question nobody asks

Ask a competent team how long their project will take. They will break the work down, estimate each block, add a margin, and tell you four months. Every step of that reasoning is sound: the breakdown is good, the people are good, the margin is reasonable.

The project will take nine months. This is not incompetence, and it is not bad luck either. It is the predictable consequence of having answered the question from the inside.

Two ways of answering the same question

Daniel Kahneman and Amos Tversky called these the inside view and the outside view. The first starts from your case: its details, your plan, the quality of your team. The second starts from elsewhere: how often have comparable situations ended one way or another?

Kahneman himself tells the story that made him understand the difference. His team was writing a textbook and estimated they needed two years. He asked a member of the group, who had seen many such projects, how long those teams had taken. The answer was: between seven and ten years, and roughly four in ten never finished. The project took eight years.

What makes the story interesting is not the estimation error. It is what followed: they had the information, it was in the room, it came from one of them, and they carried on with their original estimate. Knowing is not enough. The inside view is so vivid that it crushes a statistic, even one that has just been said out loud.

The size of the problem

Bent Flyvbjerg built the largest known database on major projects: more than sixteen thousand of them, across some twenty countries, from construction to software to the Olympic Games. The result fits in two numbers.

Remember the second bar rather than the first. Meeting budget and schedule while missing the expected benefits means delivering on time something nobody wanted. One project in two hundred escapes all three traps at once.

When you estimate your own project at four months, you are implicitly claiming membership of that 8.5%. It is possible. It requires reasons.

Building a reference class, concretely

A base rate is not a statistic you look up in a book, it is a question you construct. Three steps are enough.

1. Define the widest class that is still comparable. Not "B2B luxury e-commerce site rebuilds with a team of six", of which you will find three, and you will pick the three that suit you. Rather "site rebuilds run by an internal team". A class that is too narrow is an elegant way of returning to the inside view.
2. Find the distribution, not the average. What you want to know is not "how long it usually takes" but "what range it falls in, and what the tail looks like". In many domains the average is misleading, because extreme overruns are far more frequent than a bell curve would suggest.
3. Start from the distribution, then adjust little. The natural reflex is the reverse: start from your estimate and add a margin. Do the opposite. Take the median of the class as your starting point, and depart from it only for reasons you can name and would have named before knowing the answer you wanted.

"My case is different"

It is. Every case is different, which is what makes the objection irrefutable and therefore useless. The right question is not whether your case is different, but whether its difference is linked to the outcome.

A simple test settles it. Your difference must be nameable in advance and rare. Nameable in advance: you can write it down today, before knowing, rather than reconstructing it afterwards to explain what happened. Rare: it is not true of most projects in your class. "Our team is excellent" fails the second test, because most teams that blew their budget thought so too. "We have already delivered this exact scope twice" passes both.

When the base rate turns against you

You have to know the limits of the tool, otherwise it becomes dogma, and dogma is still a way of not thinking.

The outside view fails in three situations. When the regime has genuinely changed and past cases belong to another world: the base rates of print publishing say nothing about digital publishing. When the class is too small to produce a stable frequency, in which case admitting it beats computing a percentage on five cases. And above all, when the class is chosen after deciding the answer, which happens more often than you would think: there is always a way of slicing reality that yields the base rate you needed.

The protection against that last failing is the same as everywhere else: write your reference class down before looking for the numbers, and do not change it afterwards.

What it looks like in practice

Before your next committing decision, four questions and ten minutes.

• Which family of situations does this one belong to?
• Of the last twenty in that family, how many ended the way I hope?
• What about mine is genuinely rare within that family?
• If I am wrong, how will I see it, and by what date?

These questions do not make projects faster. They make promises more honest, which changes almost everything else: the trade-offs, the margins, and the trust people extend to you the next time.