tailspec: stress-testing AI weather models on the extremes
New AI models forecast the weather faster than physics-based systems, and just as well on the usual scorecards. Those scorecards barely touch the events that actually break things: the storm peak, the record heat, the once-a-decade cold. tailspec is an evaluation framework built to score exactly those.
result 1 · the soft cap
The single-storm trace is the ERA5 fleet-mean 100 m wind over the British farm book across the fortnight around Storm Eunice (February 2022), against each model's day-1 forecast for the same steps. The aggregate curve behind it is every 24-hour forecast in the test period, binned by the ERA5 severity band for that moment; bands run from P90-92 to ≥P99.5, so the mildest bin shown is already the top 10 per cent of the distribution.
Storm Eunice crossed southern England on 18 February 2022. Rail lines closed and about 1.4 million homes lost power. The line here is the wind the British fleet felt across the fortnight around it, as the ERA5 reanalysis settled it: the mean at 100 m over every farm cell.
Now add what two of the AI models forecast a day earlier. Both track the shape of the fortnight closely. GenCast's fan-out reaches eight of the fourteen delivery days, AIFS thirteen, so each line covers only part of the window.
The windiest step of the fortnight falls on 23 Feb, not on Eunice itself. The fleet mean reached 14.4 m s⁻¹ there and AIFS, the only model whose run reaches that day, came in 1.2 m s⁻¹ under it. At Eunice's own peak on 18 February both models landed within 0.2 m s⁻¹ of the fleet mean. One fortnight of one storm cluster fixes the direction of the error. It does not fix the size.
Widen the window to every 24-hour forecast in the test period, sorted by ERA5's severity band for that moment. Every band here is already extreme: the mildest, P90-92, is the top 10 per cent of the ERA5 distribution. Bias is smallest there and deepest in ≥P99.5, in all six models. NeuralGCM runs around twelve times deeper than the other five in that band and is drawn in its own view. The size differs sharply between them. The shape does not.
The soft cap: the bias runs negative in every severity band measured, smallest in the mildest of them (P90-92, already the top 10 per cent of the ERA5 distribution) and deepest in the rarest (≥P99.5), in all six models.
why it happens
Models trained on mean-squared error regress to the conditional mean, which truncates conditional extremes. Probabilistic scores help but do not fix it: at the desk's commitment quantile the penalty for over- and under-forecasting is asymmetric (2:1 in our wind-power backtest), so the optimal forecast sits below the ensemble mean.
Models trained to minimise average error learn that hedging towards the climatological middle is cheap insurance. Predict something close to the middle every time and the average error looks small, even when any single forecast is wrong.
A hedged forecast of an extreme looks like a blurred one: the storm is softened, the heat peak is rounded off, the cold snap is trimmed back towards normal.
The standard headline scores reward the hedge, so the failure is invisible unless you look at the tail specifically.
what tailspec measures
Does the model believe truly extreme events are possible, or does its training quietly cap them?
Generalised Pareto shape parameter ξ, fitted per grid cell by peaks-over-threshold. Too small a ξ means tails that are too thin.
When the model says a 1-in-100 event, is it actually 1-in-100?
Tail calibration after Allen et al. (2024): occurrence and severity ratios of threshold exceedances.
Do decisions made with this forecast beat the decisions you would have made without it?
Relative economic value across cost-loss ratios, plus backtested trading desks with real settlement rules (see In practice).
result 2
loading gridded data…
Peaks-over-threshold fit, per grid cell, to the exceedances above that cell's local P95 (95th-percentile) threshold. A higher ξ makes genuinely extreme values more likely, so a model whose ξ sits below ERA5's is under-weighting its own worst cases.
Weighted by grid-cell area, only GenCast runs tail-thin on the typical cell: its median tail-shape ξ (higher means heavier) sits below ERA5's, so it already believes the most extreme winds are less likely than the reanalysis record shows. AIFS, GraphCast and Pangu-Weather still average thinner overall, but the shortfall concentrates in specific regions rather than spreading evenly. CorrDiff shows no net thin-tail bias by this measure.
result 3
GenCast and AIFS share the same transformer lineage. What separates them is the training target. AIFS is trained to minimise expected error, which rewards hedging towards the climatological middle. GenCast is trained as a diffusion model, scored for producing one sharp, physically plausible outcome rather than a blurred average across many. A record-breaking event is exactly what tests that difference: can the model commit to an extreme, or does it retreat towards normal?
Paired bootstrap over 28 initialisations, resampled 10,000 times per model for the confidence intervals shown below. The GenCast-minus-AIFS difference and its p-value come from a permutation test on the same paired initialisations, so a small p-value reflects a difference too large to be shuffle noise, not just a large sample.
On record heat and cold extremes GenCast's edge is clear and significant: a Brier skill score of 0.394 against AIFS's 0.309, a difference of +0.085 (p < 0.0001). On record wind extremes GenCast leads again (+0.666, p < 0.0001), but that is failing less rather than winning outright: both models score below zero here, worse than always forecasting the climatological rate, because wind records are hard for either model to catch. On record pressure, the smoothest of the three fields, the advantage disappears: 0.599 against 0.611, statistically indistinguishable (p = 0.32).
record heat and cold
ΔBSS +0.085 [+0.058, +0.122] p < 0.0001
record wind
ΔBSS +0.666 [+0.378, +1.036] p < 0.0001
record pressure
no significant difference (p = 0.32)
GenCast's numbers make the mechanism concrete: trained to reward one sharp outcome rather than a safe average, it outscores AIFS's hedged training on record heat, cold and wind, though on wind both models still finish below the climatological rate. On the pressure field, smooth enough that there is little blur left to fix, that edge disappears too, tying the cap to the training objective rather than the model class.
next
What is this worth when money is on the line?
Two simulated trading desks, settled against what actually happened. →
The code behind this site currently lives in a private repository. If any of it bears on what you're working on, I'm happy to talk.