Your confidence says nothing unless you measure it

Two doctors look at the same scan. The first says: "it is probably benign". The second says: "there is roughly an 85% chance this is benign". Both sentences describe the same impression. Only one of them can be wrong.

That is the whole difference, and it is larger than it looks. "Probably" cannot be contradicted: if the thing happens, you were right; if it does not, you said probably, not certainly. The word is comprehensive insurance against embarrassment. A number exposes itself. And it is precisely that exposure which makes learning possible.

What being well calibrated means

A decision maker is well calibrated when the things they announce at 70% happen roughly seven times out of ten. Nothing more. It is a statistical property of your statements, not a moral quality, and it becomes measurable as soon as you have thirty or so resolved forecasts.

Note that being calibrated is not the same as being right often. You can know a great deal and be badly calibrated, announcing 95% for events that occur two times in three. That is in fact the most common expert profile, and it has a direct consequence: their recommendations are accurate, but their safety margins are too thin.

The reverse exists too. Someone answering 50% to every binary question would be perfectly calibrated and perfectly useless. Calibration alone is therefore not enough: you also need resolution, that is, the nerve to move away from 50% when you actually know something. Good judgment is both together: statements that commit, and hold.

How badly we overrate ourselves

The classic measurement asks for intervals. "Give me a range such that you are 98% sure the true value lies inside." A well calibrated person should be wrong twice in a hundred. In practice, across decades of experiments, people are wrong closer to four times in ten.

In other words, when you say "I am 98% certain", you are saying something worth about 60%. This is not a character flaw, it is the factory setting. And it does not improve on its own with experience, because nothing in a normal professional life sends the information back to you.

Why words are not enough

In 1951, an American intelligence report judged that a Soviet attack on Yugoslavia was a "serious possibility". Sherman Kent, who ran the analysis, had the curiosity to ask his colleagues what the phrase meant to them as a percentage. The answers ranged from 20 to 80%.

Everyone agreed on the sentence. Nobody agreed on what it said. That is the core problem with the vocabulary of uncertainty: "possible", "likely", "plausible", "there is a risk" are words on which each of us silently sets the dial. An entire organisation can believe it shares an assessment when it only shares an adjective.

A number can be argued with, corrected, compared. An adverb can only be interpreted.

How you calibrate

The good news is that calibration is the judgment skill that responds fastest to training. In the large forecasting tournaments of the twenty tens, about an hour of training was enough to measurably and durably reduce participants' error. One hour.

What works comes down to three habits.

1. Replace adverbs with numbers. Every time you catch yourself saying "probably", stop and put a figure on it. The first few times it will feel like false precision. It is the opposite: you have simply made visible an imprecision that was already there.
2. Write them down somewhere, with a date. An unwritten forecast will be rewritten by your memory, always in the direction that suits you. That is not dishonesty, it is how remembering works.
3. Review them in batches. Never judge a single forecast: a 90% statement that fails is not an error, it is expected one time in ten. Look at your last twenty statements at 90% and count. That is the only level at which calibration exists.

The unexpected benefit

We assume calibration serves better forecasting. It does, and that is the smallest of its benefits.

The real gain lies elsewhere: putting numbers on confidence changes the conversation. In a meeting where everyone says "I think this will work", there is nothing to debate, only convictions colliding. In a meeting where one person says 80% and another says 40%, there is suddenly a precise object: where does the gap come from, what does one of them know that the other does not, and what would have to be learned to bring them closer? Disagreement becomes usable.

And six months later, when reality has settled it, nobody can claim they saw it coming.