# RealBench: Benchmarking Data-Driven Numerical Weather Forecasting Under Operational Conditions and Extreme Event Challenges

arXiv: https://arxiv.org/abs/2605.24945 · HTML: https://mlsanity.com/paper/2605.24945/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 62 references that are in the map.
- First posted on arXiv 24 May 2026.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [Neural General Circulation Models for Weather and Climate](https://mlsanity.com/paper/2311.07222/) (2023)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere](https://mlsanity.com/paper/2306.03838/) (2023)
- [FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding](https://mlsanity.com/paper/2012.02951/) (2020)
- [GenCast: Diffusion-based ensemble forecasting for medium-range weather](https://mlsanity.com/paper/2312.15796/) (2023)
- [FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead](https://mlsanity.com/paper/2304.02948/) (2023)
- [Scaling transformer neural networks for skillful and reliable medium-range weather forecasting](https://mlsanity.com/paper/2312.03876/) (2023)
- [DYffusion: A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting](https://mlsanity.com/paper/2306.01984/) (2023)
- [ClimateLearn: Benchmarking Machine Learning for Weather and Climate Modeling](https://mlsanity.com/paper/2307.01909/) (2023)
- [EarthNet2021: A large-scale dataset and challenge for Earth surface forecasting as a guided video prediction task](https://mlsanity.com/paper/2104.10066/) (2021)
- [OneForecast: A Universal Framework for Global and Regional Weather Forecasting](https://mlsanity.com/paper/2502.00338/) (2025)
- [Kilometer-Scale Convection Allowing Model Emulation using Generative Diffusion Modeling](https://mlsanity.com/paper/2408.10958/) (2024)
- [FengWu-GHR: Learning the Kilometer-scale Medium-range Global Weather Forecasting](https://mlsanity.com/paper/2402.00059/) (2024)
- [Continuous Ensemble Weather Forecasting with Diffusion models](https://mlsanity.com/paper/2410.05431/) (2024)
- [SEEDS: Emulation of Weather Forecast Ensembles with Diffusion Models](https://mlsanity.com/paper/2306.14066/) (2023)
- [FengWu-4DVar: Coupling the Data-driven Weather Forecasting Model with 4D Variational Assimilation](https://mlsanity.com/paper/2312.12455/) (2023)
- [ClimART: A Benchmark Dataset for Emulating Atmospheric Radiative Transfer in Weather and Climate Models](https://mlsanity.com/paper/2111.14671/) (2021)
- [Towards an end-to-end artificial intelligence driven global weather forecasting system](https://mlsanity.com/paper/2312.12462/) (2023)
- [ExtremeCast: Boosting Extreme Value Prediction for Global Weather Forecast](https://mlsanity.com/paper/2402.01295/) (2024)
- [WeatherBench Probability: A benchmark dataset for probabilistic medium-range weather forecasting along with deep learning baseline models](https://mlsanity.com/paper/2205.00865/) (2022)
- [OmniCast: A Masked Latent Diffusion Model for Weather Forecasting Across Time Scales](https://mlsanity.com/paper/2510.18707/) (2025)
- [Diffusion-LAM: Probabilistic Limited Area Weather Forecasting with Diffusion](https://mlsanity.com/paper/2502.07532/) (2025)
- [IowaRain: A Statewide Rain Event Dataset Based on Weather Radars and Quantitative Precipitation Estimation](https://mlsanity.com/paper/2107.03432/) (2021)
- [Weather Prediction with Diffusion Guided by Realistic Forecast Processes](https://mlsanity.com/paper/2402.06666/) (2024)
- [KoopmanLab: machine learning for solving complex physics equations](https://mlsanity.com/paper/2301.01104/) (2023)

## Abstract (arXiv)

Accurate evaluation of weather forecasting models is critical for their reliable deployment in real-world applications. However, existing benchmarks predominantly rely on reanalysis products such as ERA5, which are generated through delayed data assimilation and do not reflect the constraints of real-time operational forecasting, thereby resulting in a systematic mismatch between benchmark performance and real-world forecasting. In this work, we introduce RealBench, a next-generation benchmark for AI weather forecasting that emphasizes realistic evaluation under operational conditions. RealBench features a strictly out-of-distribution test set spanning 2025 to eliminate data leakage and capture recent atmospheric regimes. It integrates multiple data sources, including low-latency operational analysis and a large-scale global in-situ observation dataset comprising over 10,000 stations, enabling direct evaluation against real atmospheric measurements. Beyond standard global metrics, RealBench provides a comprehensive evaluation framework for high-impact extreme events, including heatwaves, cold surges, and tropical cyclones, using event-specific metrics that better reflect real-world forecasting priorities. The evaluation results reveal substantial discrepancies between reanalysis-based metrics and real-world performance, particularly concerning extreme events. By highlighting the limitations of existing benchmarks, this work establishes a more faithful and operationally relevant evaluation paradigm, providing a rigorous foundation for advancing next-generation AI weather forecasting systems. The benchmark implementation is available at: https://github.com/lixruize-del/NWP-Benchmark.