A One-GPU Emulator at Cost Parity with ECMWF's Ensemble

the rules

Forecast, Decide, Settle

Desk A promises next-day wind power for a capacity-weighted book of German and British wind farms. It commits at the ensemble's newsvendor-optimal quantile under a 2:1 penalty for over-promising against under-promising, and delivery settles against the wind power the ERA5 reanalysis implies. Each ensemble has eight members, with the IFS ensemble subsampled to match. GraphCast and Pangu-Weather are deterministic architectures with no native ensemble, so the same quantile rule lands on their single forecast. Desk A reports what that property of the two models costs them.

Desk B quotes fair-value heating-degree-day (HDD) and cumulative-average-temperature (CAT) contracts for a four-city book each day, priced from realised-to-date ERA5 plus the bias-corrected ensemble forecast. The desk then checks whether either model can be picked off against the IFS ENS market it quotes into.

Desk A · day-ahead wind power

Every System on the Wind Book

Every day each system commits the amount of power its own ensemble says it can promise without being caught short more than a third of the time, and the book is paid out on the weather that arrived. The rungs below differ only in the forecast that went in.

Four of the six emulators land on top of the incumbent. Neither pre-registered model is measurably cheaper or dearer than the IFS ensemble on either book in either season, and GraphCast and Pangu-Weather fall in the same band. The spread across those four is a small fraction of the distance down to climatology, so on this book a desk's profit and loss cannot tell a learned model from a physics ensemble.

The matched rung cuts the IFS ensemble to the same 8 members the emulators have. More members can only sharpen its commitment quantile, so the comparison is rerun against the archive's full 50, and nothing in that group separates from that rung either.

Two of those four, GraphCast and Pangu-Weather, have no ensemble to commit from. The newsvendor quantile that shades a commitment across eight members down for the 2:1 asymmetry has nothing to select from on one member, so for them it lands on the point forecast. The extension that added them expected the price of that missing distribution to be visible, yet on the only comparison this page tests neither deterministic desk is measurably dearer than the operational ensemble. A point forecast cannot widen on the days when the ensemble would have, and 33 delivery days per book are too few for that to surface.

Which pre-registered emulator comes out ahead depends on the country, and does so consistently. AIFS ran the cheaper German book in both seasons and GenCast the cheaper British one in both, though neither ordering is large enough to matter against the interval around it. Across all four systems at parity only the German book keeps the same cheapest system, and the British one changes hands between the seasons. A league table of forecast skill will not tell you which model to put on a given book.

Of the ten systems on this book, the four that reach the incumbent are level with it on the average day. What remains is whether the average day is where the cost falls.

Desk A · where the cost falls

Cost by Realised Wind

A season mean hides which days paid for it. Sorted by how much wind actually turned up, the same delivery days show the cost of the book climbing steadily for most systems, from the calm end of the season to the windy end.

Across the 32 combinations of forecast system and book, 29 gradients point upwards and 20 are separated from flat by the bootstrap. The rise appears in every system, the operational ensemble included, and in both countries, yet no system separates from the others cleanly enough to be ranked by it.

That gradient is measured in money. A windy day puts more power on the book, so an absolute error has more room to grow whatever the forecast does. Divide each day's cost by the power actually at stake and the gradient inverts: 32 of 32 relative slopes point down, 25 of them clear of zero, so relative to the size of the job these systems are more accurate on windy days. The cost gathers at the windy end because there is more to get wrong there, which matters to a desk but does not show skill failing in the tail.

The pre-registered backtest deliberately excludes every event fortnight and every Monday initialisation, so no named storm of these two seasons is in the sample. The gradient above is therefore a gradient within ordinary weather, and the cost concentrates towards the windy end even with the genuinely extreme days taken out of the book.

Neither parity on the average day nor uneven risk across days says whether these models can represent a genuine extreme. That needs the days this backtest threw out and a measurement of the tail itself, first on a named storm and then on the shape of the whole distribution.

decision value · record events

The Ranking Depends on the Price of a Miss

A probability forecast becomes a decision only once someone names a cost: act when the forecast probability exceeds the ratio of the cost of acting to the loss of doing nothing. That ratio is different for every user, and nothing obliges the same model to win at every value of it. The explorer below prices both ensembles’ record-event forecasts across the whole spectrum.

The ordering changes with the cost ratio. On records of heat and cold, AIFS is worth more for miss-averse users and GenCast takes over above a ratio of 0.025 (interval 0.018 to 0.078). On record wind the flip is at 0.18 (0.082 to 0.20), just before the value of either ensemble runs out: past a ratio of about 0.19, neither beats climatology on wind records, the same wind difficulty the record-breaking analysis reports. On record pressure the interval spans most of the axis, so the ordering there is not resolved at all.

Which model is better at record extremes depends on the price, meaning what a miss costs relative to a false alarm. The desk above fixes one such price and pays it out in money, and this section sweeps every price.

Desk B · temperature quote game

Quoting into the Ensemble's Market

If the wind book is right that these emulators are level with the operational ensemble, a second desk quoting into a market priced off that ensemble should find nothing to arbitrage. If either model had a genuine edge over the IFS ensemble, it would show up here as profit.

Each day GenCast and AIFS post their own fair-value quote on the season's contract for a four-city book (HDD in autumn and winter, CAT in summer) into a market quoted by the IFS ENS. A desk trades only when it disagrees with the market by more than the spread, so the flat line at zero is the market itself, quoting at consensus with no edge either way.

After a realistic 2-index-point bid-ask spread, only in Autumn 21 do both desks close ahead. GenCast gives back the whole of its Winter 21/22 edge and finishes behind the market. AIFS holds through Winter 21/22 but slips behind in Summer 22, the one season it quotes alone.

No tradable edge survives the spread, as parity predicts. Across two desks and two instruments the verdict is the same, and on ordinary weather neither desk separates these emulators from the operational ensemble.