tailspec · tail-risk evaluation of AI weather models
Six AI weather emulators (GenCast, AIFS, GraphCast, Pangu-Weather, FourCastNet 3 and NeuralGCM) now run faster than the physics-based systems and match them on the usual scores. Every one of them under-forecasts extreme wind, and the rarer the event, the worse the shortfall.
a single storm
ERA5's 10 m wind, averaged over the 5 British land grid cells with the highest peak wind during the storm window - selected from ERA5 alone and held fixed across every model, so no model gets a more favourable cell. Over it, each model's 24-hour-ahead forecast for the same cells and times, from a single daily initialisation rather than an ensemble mean, so all six lines are computed the same way. Each line breaks wherever its archive stops. Provenance: 5 windiest GB-land ERA5 cells by peak 10 m wind over 2022-02-11..2022-02-24, cells fixed across all models and times; day-1 (24/36 h) leads, 00/12 UTC.
Storm Eunice crossed southern England on 18 February 2022, closing rail lines and cutting power to about 1.4 million homes. The line plotted to the rightabove is what the wind actually did at Britain's 5 windiest land cells through the fortnight around it, as the ERA5 reanalysis reconstructs it.
Now the forecasts. Each model is started afresh from that morning's atmosphere and asked what the next day holds, and the coloured lines join those 24-hour-ahead forecasts across the fortnight. All six follow the reanalysis through the whole run of storms, and each puts Eunice on the right day.
At Eunice's own peak, every model underpredicts the reanalysis, from FourCastNet 3 4.1% short to NeuralGCM 34.4% short. One storm can't prove a pattern - but the same one-sided shortfall, growing with rarity, shows up across every wind extreme in the record, next.
all wind extremes
Widen the view from one storm to every 24-hour forecast made between 2020 and 2023, worldwide, filed by how extreme the observed wind turned out to be at that place and hour. Even the mildest group on this axis is the windiest 10 per cent of the record, so nothing here is an ordinary day.
NeuralGCM's mean error is around five times the other five, so the axis has to expand to hold it.
Each forecast/observation pair is filed by the percentile the observed wind reached in ERA5's own time series at that gridcell, then reduced within the group. The groups do not overlap and the last is open-ended above P99.5. About 391,000 samples fall in each of the first four, 195,000 in P98-99 and 130,000 in each of the top two. On the mean error view, NeuralGCM is 2.1 m s⁻¹ in the mildest group and 3.7 m s⁻¹ in the rarest.
No model reaches the observation in any group, and the gap widens as the events get rarer. The five on a shared axis fall 0.21 to 0.34 m s⁻¹ short in the windiest tenth of the record and 0.51 to 0.85 m s⁻¹ short above the 99.5th percentile, while NeuralGCM does the same thing an order of magnitude larger. What all six share is the shape rather than the size, a shortfall that grows with rarity in every one of them.
the mechanism
the technical version
A point loss is minimised by a functional of the conditional distribution rather than by a draw from it. Squared error is minimised by the conditional mean E[y | x] and absolute error by the conditional median, and either way p(y | x) collapses to one number per cell. A field of those numbers varies less than a draw does, by the law of total variance, and the gap scales with the conditional spread, which is what compresses conditional extremes. A proper score for the whole distribution lifts that constraint on the individual members but not on anything read off the ensemble mean. At a decision threshold the loss is asymmetric anyway (2:1 in our wind-power backtest), so the cost-minimising commitment is a quantile of the predictive distribution and not its centre.
Most of these emulators are trained on a point loss, and which point loss barely matters. GraphCast and AIFS minimise squared error, whose optimum is the conditional mean. Pangu-Weather minimises absolute error, whose optimum is the conditional median. Either way the target is a summary of every atmosphere still consistent with this morning's observations rather than one of those atmospheres, and a summary falls furthest from a real draw exactly where those atmospheres disagree most, at the top of the distribution.
Training on a probabilistic score loosens this without undoing it. FourCastNet 3 is trained on a CRPS objective and GenCast is a diffusion model that draws samples rather than averaging over them, and both still come in under the observation in every group on the panel above. Some of that is where the number gets read, since headline products are usually taken off the ensemble mean, which puts the averaging straight back in. And where a real decision sits the penalties are rarely symmetric, so what minimises cost is a quantile rather than the middle.
What keeps this invisible is arithmetic. Global RMSE is dominated by the ordinary hours that make up almost all of the record, so a shortfall confined to the rarest per cent barely moves it. The tail is not being measured, and it stays that way until it is scored on its own.
what tailspec measures
Does the model's own distribution reach the most extreme events, or does it thin out too fast?
Generalised Pareto shape parameter ξ, fitted per grid cell by peaks-over-threshold. Too small a ξ means tails that are too thin.
When the model says a 1-in-100 event, is it actually 1-in-100?
Tail calibration after Allen et al. (2024). The occurrence ratio compares how often a threshold was actually exceeded with how often the forecast said it would be. The severity ratio compares how far past the threshold those exceedances went.
What a decision taken on this forecast costs, against the same decision taken without it.
Relative economic value across cost-loss ratios, plus backtested trading desks with real settlement rules (see In practice).
tail shape
loading gridded data…
the technical version
Peaks-over-threshold fit, per grid cell, to the exceedances above that cell's local P95 (95th-percentile) threshold. The more positive ξ is, the heavier the tail, so a model whose ξ falls short of ERA5's is under-weighting its own worst cases.
GenCast is the only forecast model whose tail shape ξ falls short of ERA5's at the median grid cell, which makes its own most extreme winds rarer than the reanalysis record shows. AIFS, GraphCast and Pangu-Weather come in under ERA5 on the area-weighted mean but not at the median, so their shortfall concentrates in particular regions rather than spreading evenly. CorrDiff is the one exception in the other direction, heavier-tailed than ERA5 on both. It is a diffusion downscaler rather than a forecast model, so no squared-error objective is pulling its output towards a conditional mean.
record-breaking events
GenCast and AIFS come out of the same transformer lineage. What separates them is what they were trained to do. AIFS minimises mean squared error, and the forecast that minimises squared error is the conditional mean. GenCast is a conditional diffusion model, trained by denoising rather than by error minimisation and scored with CRPS, which rewards a single sharp atmosphere that could plausibly have happened over a blurred average of many that could not. A record-breaking event tests that difference directly, because the model has to either commit to a value beyond the standing record or fall back towards normal.
the technical version
Paired bootstrap over 28 initialisations, resampled 10,000 times per model for the intervals shown below. The GenCast-minus-AIFS difference and its p-value come from a permutation test on the same paired initialisations, so a small p-value reflects a difference too large to be shuffle noise, not just a large sample.
Records of heat and cold separate the two clearly. GenCast reaches a Brier skill of 0.394 to AIFS's 0.309, a difference of +0.085 (p < 0.0001). Wind records are hard for both. GenCast leads there by +0.666 (p < 0.0001), but both land below zero, so both do worse than simply quoting the long-run rate at which wind records fall. Mean sea-level pressure, the smoothest of the three fields, shows no advantage either way, 0.599 to 0.611 and indistinguishable at p = 0.32.
record heat & cold
GenCast ahead by +0.085 [+0.058, +0.122] p < 0.0001
record wind
GenCast ahead by +0.666 [+0.378, +1.036] p < 0.0001
record pressure
no separation (p = 0.32)
GenCast and AIFS differ in objective and share a lineage, so the gap between them isolates the objective rather than the architecture. How large that gap is depends on the field. It is widest on wind (+0.666, though from a baseline where neither model beats the climatological rate), clear and significant on heat and cold (+0.085), and absent on pressure, the field with least blur to remove.
next
The same forecasts, priced on a trading desk
Two simulated desks run on GenCast and AIFS, then paid out on the weather that actually arrived. →
The code behind this site is in a private repository for now. If any of it is useful to you, get in touch.