# WeatherReal: A Benchmark Based on In-Situ Observations for Evaluating Weather Models

arXiv: https://arxiv.org/abs/2409.09371 · HTML: https://mlsanity.com/paper/2409.09371/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 42 references that are in the map.
- First posted on arXiv 14 Sep 2024.

## Builds on (most cited first, 10 of 10)

- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [Skillful Precipitation Nowcasting using Deep Generative Models of Radar](https://mlsanity.com/paper/2104.00954/) (2021)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [Forecasting Global Weather with Graph Neural Networks](https://mlsanity.com/paper/2202.07575/) (2022)
- [Sub-seasonal forecasting with a large ensemble of deep-learning weather prediction models](https://mlsanity.com/paper/2102.05107/) (2021)
- [Deep Learning for Day Forecasts from Sparse Observations](https://mlsanity.com/paper/2306.06079/) (2023)
- [DiffDA: a Diffusion Model for Weather-scale Data Assimilation](https://mlsanity.com/paper/2401.05932/) (2024)
- [FengWu-4DVar: Coupling the Data-driven Weather Forecasting Model with 4D Variational Assimilation](https://mlsanity.com/paper/2312.12455/) (2023)
- [Data driven weather forecasts trained and initialised directly from observations](https://mlsanity.com/paper/2407.15586/) (2024)

## Built on by (most cited first, 8 of 8)

- [OMG-HD: A High-Resolution AI Weather Model for End-to-End Forecasts from Observations](https://mlsanity.com/paper/2412.18239/) (2024)
- [EPT-2 Technical Report](https://mlsanity.com/paper/2507.09703/) (2025)
- [Extreme Weather Bench: A framework and benchmark for evaluation of high-impact weather](https://mlsanity.com/paper/2605.01126/) (2026)
- [Skillful high-resolution weather forecasting independent of physical models](https://mlsanity.com/paper/2605.28153/) (2026)
- [PeakWeather: MeteoSwiss Weather Station Measurements for Spatiotemporal Deep Learning](https://mlsanity.com/paper/2506.13652/) (2025)
- [RealBench: Benchmarking Data-Driven Numerical Weather Forecasting Under Operational Conditions and Extreme Event Challenges](https://mlsanity.com/paper/2605.24945/) (2026)
- [Do AI weather models miss extremes?](https://mlsanity.com/paper/2608.09972/) (2026)
- [A Station-Based Evaluation of Machine Learning-based Weather Forecasting Models in Northern Norway](https://mlsanity.com/paper/2609.10564/) (2026)

## Abstract (arXiv)

In recent years, AI-based weather forecasting models have matched or even outperformed numerical weather prediction systems. However, most of these models have been trained and evaluated on reanalysis datasets like ERA5. These datasets, being products of numerical models, often diverge substantially from actual observations in some crucial variables like near-surface temperature, wind, precipitation and clouds - parameters that hold significant public interest. To address this divergence, we introduce WeatherReal, a novel benchmark dataset for weather forecasting, derived from global near-surface in-situ observations. WeatherReal also features a publicly accessible quality control and evaluation framework. This paper details the sources and processing methodologies underlying the dataset, and further illustrates the advantage of in-situ observations in capturing hyper-local and extreme weather through comparative analyses and case studies. Using WeatherReal, we evaluated several data-driven models and compared them with leading numerical models. Our work aims to advance the AI-based weather forecasting research towards a more application-focused and operation-ready approach.