# Validating Deep Learning Weather Forecast Models on Recent High-Impact Extreme Events

arXiv: https://arxiv.org/abs/2404.17652 · HTML: https://mlsanity.com/paper/2404.17652/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 76 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Builds on (most cited first, 24 of 24)

- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [Neural General Circulation Models for Weather and Climate](https://mlsanity.com/paper/2311.07222/) (2023)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [GenCast: Diffusion-based ensemble forecasting for medium-range weather](https://mlsanity.com/paper/2312.15796/) (2023)
- [FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead](https://mlsanity.com/paper/2304.02948/) (2023)
- [Scaling transformer neural networks for skillful and reliable medium-range weather forecasting](https://mlsanity.com/paper/2312.03876/) (2023)
- [Sub-seasonal forecasting with a large ensemble of deep-learning weather prediction models](https://mlsanity.com/paper/2102.05107/) (2021)
- [Deep Learning for Day Forecasts from Sparse Observations](https://mlsanity.com/paper/2306.06079/) (2023)
- [Latent diffusion models for generative precipitation nowcasting with accurate uncertainty quantification](https://mlsanity.com/paper/2304.12891/) (2023)
- [Engression: Extrapolation through the Lens of Distributional Regression](https://mlsanity.com/paper/2307.00835/) (2023)
- [Uncertainty quantification for data-driven weather models](https://mlsanity.com/paper/2403.13458/) (2024)
- [Forecaster's Dilemma: Extreme Events and Forecast Evaluation](https://mlsanity.com/paper/1512.09244/) (2015)
- [Neural Networks for Extreme Quantile Regression with an Application to Forecasting of Flood Risk](https://mlsanity.com/paper/2208.07590/) (2022)
- [Gradient boosting for extreme quantile regression](https://mlsanity.com/paper/2103.00808/) (2021)
- [Global Extreme Heat Forecasting Using Neural Weather Models](https://mlsanity.com/paper/2205.10972/) (2022)
- [Do AI models produce better weather forecasts than physics-based models? A quantitative evaluation case study of Storm Ciar'an](https://mlsanity.com/paper/2312.02658/) (2023)
- [Evaluating forecasts for high-impact events using transformed kernel scores](https://mlsanity.com/paper/2202.12732/) (2022)
- [Regression modelling of spatiotemporal extreme U.S. wildfires via partially-interpretable neural networks](https://mlsanity.com/paper/2208.07581/) (2022)
- [Extremal Random Forests](https://mlsanity.com/paper/2201.12865/) (2022)
- [Deep graphical regression for jointly moderate and extreme Australian wildfires](https://mlsanity.com/paper/2308.14547/) (2023)
- [On some limitations of data-driven weather forecasting models](https://mlsanity.com/paper/2309.08473/) (2023)

## Built on by (most cited first, 14 of 14)

- [Can AI weather models predict out-of-distribution gray swan tropical cyclones?](https://mlsanity.com/paper/2410.14932/) (2024)
- [Huge Ensembles Part I: Design of Ensemble Weather Forecasts using Spherical Fourier Neural Operators](https://mlsanity.com/paper/2408.03100/) (2024)
- [Numerical models outperform AI weather forecasts of record-breaking extremes](https://mlsanity.com/paper/2508.15724/) (2025)
- [Binned Spectral Power Loss for Improved Prediction of Chaotic Systems](https://mlsanity.com/paper/2502.00472/) (2025)
- [Progression: an extrapolation principle for regression](https://mlsanity.com/paper/2410.23246/) (2024)
- [Constructing Extreme Heatwave Storylines with Differentiable Climate Models](https://mlsanity.com/paper/2506.10660/) (2025)
- [Turning Up the Heat: Assessing 2-m Temperature Forecast Errors in AI Weather Prediction Models During Heat Waves](https://mlsanity.com/paper/2504.21195/) (2025)
- [Learning vertical coordinates via automatic differentiation of a dynamical core](https://mlsanity.com/paper/2512.17877/) (2025)
- [Extrapolation in Statistical Learning with Extreme Value Theory](https://mlsanity.com/paper/2605.01909/) (2026)
- [Breaking the Statistical Similarity Trap in Extreme Convection Detection](https://mlsanity.com/paper/2509.09195/) (2025)
- [Native Extrapolation Awareness in Flow-Based Conditional Generation](https://mlsanity.com/paper/2602.13061/) (2026)
- [Out-of-Distribution generalization of quantile regression with heavy tailed inputs: an SVM approach](https://mlsanity.com/paper/2606.00265/) (2026)
- [Integrating GNSS-Derived Zenith Wet Delay into a Weather Foundation Model Improves Precipitation Forecasting](https://mlsanity.com/paper/2607.05658/) (2026)
- [A Station-Based Evaluation of Machine Learning-based Weather Forecasting Models in Northern Norway](https://mlsanity.com/paper/2609.10564/) (2026)

## Abstract (arXiv)

The forecast accuracy of machine learning (ML) weather prediction models is improving rapidly, leading many to speak of a "second revolution in weather forecasting". With numerous methods being developed and limited physical guarantees offered by ML models, there is a critical need for a comprehensive evaluation of these emerging techniques. While this need has been partly fulfilled by benchmark datasets, they provide little information on rare and impactful extreme events or on compound impact metrics, for which model accuracy might degrade due to misrepresented dependencies between variables. To address these issues, we compare ML weather prediction models (GraphCast, PanguWeather, and FourCastNet) and ECMWF's high-resolution forecast system (HRES) in three case studies: the 2021 Pacific Northwest heatwave, the 2023 South Asian humid heatwave, and the North American winter storm in 2021. We find that ML weather prediction models locally achieve similar accuracy to HRES on the record-shattering Pacific Northwest heatwave but underperform when aggregated over space and time. However, they forecast the compound winter storm substantially better. We also highlight structural differences in how the errors of HRES and the ML models build up to that event. The ML forecasts lack important variables for a detailed assessment of the health risks of the 2023 humid heatwave. Using a possible substitute variable, prediction errors show spatial patterns with the highest danger levels over Bangladesh being underestimated by the ML models. Generally, case-study-driven, impact-centric evaluation can complement existing research, increase public trust, and aid in developing reliable ML weather prediction models.