tailspec · tail-risk evaluation of AI weather models
Six AI weather emulators (GenCast, AIFS, GraphCast, Pangu-Weather, FourCastNet 3 and NeuralGCM) now run faster than the physics-based systems and match them on the usual scores. Every one of them under-forecasts extreme wind, and the rarer the event, the worse the shortfall.
a single storm
ERA5's 100 m wind, averaged over every 0.25° grid cell holding a British wind farm, which smooths gusts, so this is not a gust trace. Over it, each model's 24-hour-ahead forecast for the same times, averaged across its ensemble members. GenCast and AIFS are the two emulators run here with a daily-initialised ensemble, which is what a day-by-day line needs. The other four sit on a five-day init spacing, too sparse to draw one, and appear in the aggregate below. Each line breaks wherever its archive stops. Provenance: farm-cell mean, hub-height 100 m, alpha=0.14; day-1 (24/36 h) leads.
Storm Eunice crossed southern England on 18 February 2022, closing rail lines and cutting power to about 1.4 million homes. The line plotted to the rightabove is what the wind actually did at turbine height over Britain's wind farms through the fortnight around it, as the ERA5 reanalysis reconstructs it.
Now the forecasts. Each model is started afresh from that morning's atmosphere and asked what the next day holds, and the coloured lines join those 24-hour-ahead forecasts up across the fortnight. Both follow the reanalysis through the whole run of storms, and both put Eunice on the right day.
At Eunice's own peak both models land within 0.2 m s⁻¹ of the reanalysis. One storm can show that a model sees an event of this kind coming. It cannot show whether the model shaves the top off one, and answering that takes every extreme in the record.
all wind extremes
Widen the view from one storm to every 24-hour forecast made between 2020 and 2023, worldwide, filed by how extreme the observed wind turned out to be at that place and hour. Even the mildest group on this axis is the windiest 10 per cent of the record, so nothing here is an ordinary day.
NeuralGCM's shortfall is around 12 times the other five, so the axis has to expand to hold it.
Each forecast/observation pair is filed by the percentile the observed wind reached in ERA5's own distribution for that place, then averaged within the group. The groups do not overlap and the last is open-ended above P99.5. About 381,000 samples fall in each of the first four, 190,000 in P98-99 and 95,000 in each of the top two. NeuralGCM falls 2.2 m s⁻¹ short in the mildest group and 5.3 m s⁻¹ short in the rarest.
No model reaches the observation in any group, and the gap widens as the events get rarer. The five on a shared axis fall 0.07 to 0.22 m s⁻¹ short in the windiest tenth of the record and 0.26 to 0.63 m s⁻¹ short above the 99.5th percentile, while NeuralGCM does the same thing an order of magnitude larger. What all six share is the shape rather than the size, a ceiling on how far a forecast will go that firms up as the event gets rarer.
the mechanism
the technical version
Under squared-error loss the optimal prediction is the conditional mean E[y | x], so a deterministic emulator trained on MSE converges towards it and compresses conditional extremes. Probabilistic training restores sharpness in individual members, but any product read off the ensemble mean inherits the compression. At a decision threshold the loss is asymmetric anyway (2:1 in our wind-power backtest), so the commitment that minimises cost is not the mean either.
Almost every one of these emulators is trained to minimise mean squared error. The forecast that minimises squared error is the conditional mean, which is not a weather field anyone will ever observe. It is an average over every atmosphere consistent with this morning's observations, and averaging costs most where those atmospheres disagree most, at the top of the distribution.
Training on a probabilistic score softens this rather than removing it. Individual members come out sharp, but the headline numbers are still read off the ensemble mean, which carries the averaging straight back in. And where a real decision sits the penalties are rarely symmetric, so even a well-calibrated forecast can be commercially wrong when it is read off the middle.
What keeps this invisible is arithmetic. Global RMSE is dominated by the ordinary hours that make up almost all of the record, so a shortfall confined to the rarest per cent barely moves it. The tail is not being measured, and it stays that way until it is scored on its own.
what tailspec measures
Does the model's own distribution reach the most extreme events, or does it thin out too fast?
Generalised Pareto shape parameter ξ, fitted per grid cell by peaks-over-threshold. Too small a ξ means tails that are too thin.
When the model says a 1-in-100 event, is it actually 1-in-100?
Tail calibration after Allen et al. (2024). The occurrence ratio compares how often a threshold was actually exceeded with how often the forecast said it would be. The severity ratio compares how far past the threshold those exceedances went.
What a decision taken on this forecast costs, against the same decision taken without it.
Relative economic value across cost-loss ratios, plus backtested trading desks with real settlement rules (see In practice).
tail shape
loading gridded data…
the technical version
Peaks-over-threshold fit, per grid cell, to the exceedances above that cell's local P95 (95th-percentile) threshold. The more positive ξ is, the heavier the tail, so a model whose ξ falls short of ERA5's is under-weighting its own worst cases.
GenCast is the only forecast model whose tail shape ξ falls short of ERA5's at the median grid cell, which makes its own most extreme winds rarer than the reanalysis record shows. AIFS, GraphCast and Pangu-Weather come in under ERA5 on the area-weighted mean but not at the median, so their shortfall concentrates in particular regions rather than spreading evenly. CorrDiff is the one exception in the other direction, heavier-tailed than ERA5 on both. It is a diffusion downscaler rather than a forecast model, so no squared-error objective is pulling its output towards a conditional mean.
record-breaking events
the technical version
Paired bootstrap over 28 initialisations, resampled 10,000 times per model for the intervals shown below. The GenCast-minus-AIFS difference and its p-value come from a permutation test on the same paired initialisations, so a small p-value reflects a difference too large to be shuffle noise, not just a large sample.
GenCast and AIFS come out of the same transformer lineage. What separates them is what they were trained to do. AIFS minimises mean squared error, and the forecast that minimises squared error is the conditional mean. GenCast is a conditional diffusion model, trained by denoising rather than by error minimisation and scored with CRPS, which rewards a single sharp atmosphere that could plausibly have happened over a blurred average of many that could not. A record-breaking event puts that difference under load, because the model has to either commit to a value beyond the standing record or fall back towards normal.
Records of heat and cold separate the two clearly. GenCast reaches a Brier skill of 0.394 to AIFS's 0.309, a difference of +0.085 (p < 0.0001). Wind records are hard for both. GenCast leads there by +0.666 (p < 0.0001), but both land below zero, so both do worse than simply quoting the long-run rate at which wind records fall. Mean sea-level pressure, the smoothest of the three fields, shows no advantage either way, 0.599 to 0.611 and indistinguishable at p = 0.32.
record heat & cold
GenCast ahead by +0.085 [+0.058, +0.122] p < 0.0001
record wind
GenCast ahead by +0.666 [+0.378, +1.036] p < 0.0001
record pressure
no separation (p = 0.32)
GenCast and AIFS differ in objective and share a lineage, so the gap between them isolates the objective rather than the architecture. How large that gap is depends on the field. It is widest on wind (+0.666), though from a baseline where neither model beats the climatological rate, clear and significant on heat and cold (+0.085), and absent on pressure, the field with least blur to remove.
next
The same forecasts, priced on a trading desk
Two simulated desks run on GenCast and AIFS, then paid out on the weather that actually arrived. →
The code behind this site is in a private repository for now. If any of it bears on what you're working on, I'm happy to talk.