tailspec · tail-risk evaluation of AI weather models
Six AI weather emulators (GenCast, AIFS, GraphCast, Pangu-Weather, FourCastNet 3 and NeuralGCM) now run faster than the physics-driven models and match them on the usual scores. Each one of them under-predicts extremes. The rarer the event, the worse the shortfall in prediction.
a single storm
ERA5's 10 m wind, averaged over the 5 British land cells the storm hit hardest. Each model's 24-hour-ahead forecast for the same cells and times is overlaid, from a single daily run rather than an ensemble mean.
Storm Eunice crossed southern England on 18 February 2022, closing rail lines and cutting power to about 1.4 million homes. The line plotted to the rightabove is what the wind actually did in the locations the storm hit hardest, as the ERA5 reanalysis reconstructs it.
Now let's look at the AI forecasts. Each model is started fresh from the morning atmospheric condition and asked what the next day holds. The coloured lines join those 24-hour-ahead forecasts across the fortnight, and all six track the reanalysis through the whole run of storms.
At Eunice's own peak, every model underpredicts the actual conditions, by an average of 10.9% and NeuralGCM alone falls 34.4% short. One storm can't prove a pattern, but consistent one-sided behaviour will show up if we consider every wind extreme in the record.
all wind extremes
Broaden the view from one storm to every 24-hour forecast made between 2020 and 2023, worldwide, filed by how extreme the observed wind turned out to be at that place and hour. Even the mildest group on this axis is the windiest 10 per cent of the record, so nothing here is an ordinary day.
Each forecast/observation pair is filed by the percentile the observed wind reached in ERA5's own time series at that gridcell, then reduced within the group. About 391,000 samples fall in each of the first four, 195,000 in P98-99 and 130,000 in each of the top two.
No model reaches the observation in any extreme group, and the gap widens as the events get rarer. The five on a shared axis fall 0.21 to 0.85 m s⁻¹ short, worse in the rarer bands than the milder ones, while NeuralGCM falls short by an order of magnitude more. What all six share is the trend: a systematic underprediction that grows with rarity.
the mechanism
the technical version
A point loss is minimised by a functional of the conditional distribution, not a draw from it: squared error by the conditional mean E[y | x], absolute error by the conditional median, either way collapsing p(y | x) to one number per cell. By the law of total variance, a field of those numbers varies less than a field of real draws would, by a gap that scales with the conditional spread, which is what compresses conditional extremes. A proper score for the whole distribution lifts that constraint on individual samples, but not on anything read off their ensemble mean. At a decision threshold the loss is asymmetric anyway (2:1 in our wind-power backtest), so the cost-minimising commitment is a quantile of the predictive distribution, not its centre.
Most of these AI models are trained on a point loss. GraphCast and AIFS minimise squared error, whose optimum is the conditional mean, and Pangu-Weather minimises absolute error, whose optimum is the conditional median. What matters is not which point loss, but that it is a point loss at all: either way the target is a summary of every atmosphere still consistent with this morning's observations rather than one of those atmospheres, and a summary falls furthest from a real draw exactly where those atmospheres disagree most, at the top of the distribution.
Training on a probabilistic score eases the constraint but does not remove it. FourCastNet 3 is trained on a CRPS objective and GenCast is a diffusion model that draws samples rather than averaging over them, and both still fall short of the observation in every group on the panel above. This is likely partially due to where the number gets read: headline products are usually taken off the ensemble mean, which puts the averaging back in. And real decisions rarely carry symmetric penalties, so what minimises cost is a quantile of the forecast, not the middle of it.
Global RMSE weights every sample by how often it occurs, not by how much a miss there costs. The rarest per cent of the record barely moves it, however large the shortfall, because the ordinary hours that make up the rest of the record dominate the average by sheer count. The tail is not being measured, and it will not be until it is scored on its own terms, which is what tailspec does.
what tailspec measures
Does the model's own distribution reach the most extreme events, or does it thin out too fast?
Generalised Pareto shape parameter ξ, fitted per grid cell by peaks-over-threshold. Too small a ξ means tails that are too thin.
When the model says a 1-in-100 event, is it actually 1-in-100?
Tail calibration after Allen et al. (2024): an occurrence ratio (exceedances observed against exceedances forecast) and a severity ratio (how far past the threshold, once exceeded).
What a decision made using this forecast is worth, against the same decision made without it.
Relative economic value across cost-loss ratios, plus backtested trading desks with real settlement rules (see In practice).
tail shape
loading gridded data…
the technical version
Peaks-over-threshold fit, per grid cell, to the exceedances above that cell's local P95 (95th-percentile) threshold. The more positive ξ is, the heavier the tail, so a model whose ξ falls short of ERA5's is under-weighting its own worst cases.
GenCast is the only forecast model with a thinner tail than ERA5 at the median grid cell, understating its own most extreme winds almost everywhere. AIFS, GraphCast and Pangu-Weather fall short of ERA5 on average but not at the median, so their shortfall concentrates in particular regions rather than spreading evenly. CorrDiff runs the other way, heavier-tailed than ERA5 throughout — unsurprising for a diffusion downscaler, with no squared-error objective pulling its output towards a conditional mean.
tail calibration
Tail heaviness asks whether a model's own distribution reaches its most extreme values. Calibration asks a different question: when the model says a threshold has a given chance of being exceeded, is that chance actually right? Allen et al. (2024) split the answer into two pieces: an occurrence ratio (is the exceedance probability itself correct) and a severity ratio (given exceedance, is the size of the overshoot correct). The diagnostic needs full ensemble output, not a single trajectory, so the comparison here is GenCast and AIFS at full member resolution against the IFS ENS physics ensemble, all three restricted to the same 28 initialisation dates.
the technical version
Reliability curve at the region's 99th-percentile threshold: the empirical CDF of the excess PIT values among exceedances, over u ∈ [0, 1]. A tail-calibrated forecast's curve lies on the diagonal. The occurrence ratio is exceedances observed divided by exceedance probability summed over all forecasts; above 1 means the forecast under-states how often the threshold is crossed, below 1 means it over-states it.
Occurrence ratio at this threshold (1.00 = exactly the observed exceedance rate): GenCast 1.05, AIFS 0.91, IFS ENS 0.62.
For every exceedance of the region's 99th-percentile threshold, the excess PIT value (where in its own predicted distribution the observed excess falls) is pooled into an empirical CDF over u. A calibrated forecast's curve sits on the diagonal. Ensemble-only: GenCast and AIFS run at full member resolution here, not the single-trajectory archive the rest of the page uses, scored against the IFS ENS physics ensemble restricted to the same 28 dates so all three see identical thresholds.
Calibration holds up reasonably at a day ahead and erodes with lead time, the same direction as every other result on this page. The clearest break is wind in the Pacific Northwest at the longest lead, where both AI ensembles under-state how often the threshold is crossed by more than the physics ensemble does: the extremes arrive more often than either AI model's own distribution admits.
record-breaking events
GenCast and AIFS share a closely related architecture, both built on the graph-based encode-process-decode design of GraphCast. What separates them is what they were trained to do. AIFS minimises mean squared error, and the forecast that minimises squared error is the conditional mean. GenCast is a conditional diffusion model, trained by denoising rather than by error minimisation and scored with CRPS, which rewards a single sharp atmosphere that could plausibly have happened over a blurred average of many that could not. A record-breaking event tests this difference directly, because the model has to either commit to an extreme beyond the standing record or fall back towards an average value.
the technical version
Paired bootstrap over 28 initialisations, resampled 10,000 times per model for the intervals shown below. The GenCast-minus-AIFS difference and its p-value come from a permutation test on the same paired initialisations.
Records of heat and cold separate the two clearly, with GenCast ahead on Skill at calling a record before it happens, relative to simply quoting the long-run rate at which records of that kind fall. Zero is that baseline. Below it, a forecast does worse than the naive rate.. Wind records are hard for both models: GenCast still leads, but neither beats the long-run rate at which wind records fall. Mean sea-level pressure, the smoothest of the three fields, shows no advantage either way.
record heat & cold
GenCast ahead by +0.085 [+0.058, +0.122] p < 0.0001
record wind
GenCast ahead by +0.666 [+0.378, +1.036] p < 0.0001
record pressure
no separation (p = 0.32)
GenCast and AIFS differ in objective and share a lineage, so the gap between them isolates the objective rather than the architecture. How large that gap is depends on the field: widest on wind, clear on heat and cold, and absent on pressure, the field with the least blur to remove.
next
The same forecasts, priced on a trading desk
Two simulated desks run on GenCast and AIFS, then paid out on the weather that actually arrived. →
The code behind this site is in a private repository for now. If any of it is useful to you, get in touch.