An idea that sounds right can measure worse: why the median lost to the mean
At Isobaria, swapping the weighted mean for the median seemed like the obvious improvement for the confidence index. Measured against 151,000 hours of ERA5, it was worse. The story of why, and what we did instead.
Isobaria shows a consensus among three weather models: ECMWF, ICON and GFS. With 35 °C, 36 °C and 30 °C, the weighted mean gave 33.9 °C. A figure that none of the three models actually stands behind, and one that also lands a full degree below what two of them say. The dissenter — the model that strays from the pack — pulled the result toward itself, even as a minority of one.
The fix we thought of first is the one anyone would think of: use the median. It's the textbook remedy against outliers, and in this specific case it would have given 35 °C, exactly what two of the three models agree on. So we implemented it, documented it in an ADR, and called it "weighted median, always."
Measured against reality, it was worse than what it replaced.
How you measure something like this
Isobaria calibrates its weights by comparing what each model predicted, anywhere from one to seven days ahead, against what actually happened. That's possible thanks to Open-Meteo's Previous Runs API, which keeps a record of what each model said at every past instant, checked against ERA5, the ECMWF's reanalysis dataset. There was no need to wait months accumulating our own predictions: the historical record was already there, so the comparison ran over 151,000 hours.
The result, with the median applied across all hours versus the previous algorithm:
| Variant | Temperature | Wind | Spread (wind) |
|---|---|---|---|
| Before the change | 1.257 °C | 2.781 km/h | ×2.31 |
| Median always (1st version) | 1.269 °C | 2.831 km/h | ×2.41 |
Worse on temperature (−0.9%) and worse on wind (−1.9%). The intuition behind the change was reasonable — protect the consensus from a dissenter — and the measurement disproved it just the same.
Why it loses, specifically
The reason, once you see it, is obvious, and that's exactly why it's easy to miss: with three models, the median always returns the value of one of them. It's not an average of three opinions, it's picking the middle one and throwing the other two in the trash.
That's exactly what you want when there's a genuine dissenter. The problem is that a genuine dissenter isn't the normal case. The 35 · 36 · 30 case is the exception; the typical case is that all three models roughly agree, with small fractions of a degree between them. There, the median still discards two-thirds of the information — 86% of the hours, according to the comparison — to keep a single model that, for that particular hour, has no reason to be more trustworthy than the other two.
The weighted mean, when all three agree, does use the information from all three. Switching from median to "always" didn't fix the rare case without breaking the normal one; it fixed it at the cost of breaking it.
The hybrid, and what it cost not to fabricate consensus
The solution that stuck isn't "median or mean," it's a condition:
centro = media ponderada si < 3 modelos, o si ningún modelo se sale de tolerancia
= mediana ponderada si algún modelo está fuera de tolerancia
You pay for the median only when there's something worth paying for. The result, in the same comparison:
| Variant | Temperature | Wind | Spread (wind) |
|---|---|---|---|
| Before the change | 1.257 °C | 2.781 km/h | ×2.31 |
| Median always (1st version) | 1.269 °C | 2.831 km/h | ×2.41 |
| Hybrid (current) | 1.259 °C | 2.770 km/h | ×2.42 |
It beats the previous algorithm on wind (+0.40%), ties on temperature (two thousandths of a degree), and keeps the property that motivated all of this in the first place: the screen never shows a figure that no model actually holds.
There's a stricter constraint underneath this decision, and it's the one that really governs it: a model that falls outside tolerance loses control over the value, but not its vote on whether there's agreement or not. If, on top of not counting toward the center, it were also erased from the calculation of how much the models coincide, the agreement index could read "the models agree" in exactly the case where one of the three is looking at a different day entirely. That's why a model outside tolerance still scores zero on its share of the agreement and still triggers an explicit warning about which model disagrees and by how much, even if the overall number doesn't drop below any threshold. Optimizing only the displayed value, without that second piece, would have been fabricating consensus where none exists.
What this doesn't fix, stated plainly
Neither version covers precipitation probability. Its tolerance is 25 percentage points, so a 26% against a center of 3% doesn't count as dissent, and the mean wins out — pulled upward by the highest model. The median would have given the 3% there. It's a threshold chosen by judgment, not measured, because Isobaria's calibration today covers precipitation, temperature and wind, but not the probability of rain. And the reference all of this is measured against, ERA5, is produced by the ECMWF: the same verification report flags it when that model wins every row in the comparison, instead of staying quiet about it.
The hybrid's accuracy gain, besides, is small: tenths of a percentage point in temperature, less than half a point in wind. What really changes isn't how much better it gets, but that now there's a number to know how much you're getting right, and that number lets you reject an idea that sounds reasonable but measures worse — which is exactly what happened to the first version of this very change.
What we take from this
A correct intuition about the problem doesn't guarantee the correct fix. That the median ignores outliers is true, and it's exactly what was needed for the dissenter case. Applying it always solved 14% of the hours at the cost of the remaining 86%, and that proportion only shows up if you measure it.
Measuring against something real, not against your own hypothesis, is what exposes the mistake. The Previous Runs history against ERA5 already existed; there was no need to wait months to discover the first version was worse, just a need to compare against data that hadn't been generated to confirm it.
Taking control away from a value and taking away its vote are two different decisions, and confusing them fabricates the appearance of agreement. A model that disagrees stops setting the displayed figure, but it still has to count toward whether the system says "the models agree" — otherwise the confidence index could go up precisely when it should go down.
If you're designing an index that combines several sources that don't always agree with each other, write to me: the expensive part is almost never the formula, it's measuring it against the truth before trusting the intuition.