# Do AI models produce better weather forecasts than physics-based models? A quantitative evaluation case study of Storm Ciar'an

arXiv: https://arxiv.org/abs/2312.02658 · HTML: https://mlsanity.com/paper/2312.02658/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 60 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 14 of 14)

- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](https://mlsanity.com/paper/2111.13587/) (2021)
- [Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere](https://mlsanity.com/paper/2306.03838/) (2023)
- [Towards Explainable Artificial Intelligence](https://mlsanity.com/paper/1909.12072/) (2019)
- [Neural networks for post-processing ensemble weather forecasts](https://mlsanity.com/paper/1805.09091/) (2018)
- [Sub-seasonal forecasting with a large ensemble of deep-learning weather prediction models](https://mlsanity.com/paper/2102.05107/) (2021)
- [Deep Learning for Day Forecasts from Sparse Observations](https://mlsanity.com/paper/2306.06079/) (2023)
- [Deep Learning for Post-Processing Ensemble Weather Forecasts](https://mlsanity.com/paper/2005.08748/) (2020)
- [Discovering Causal Relations and Equations from Data](https://mlsanity.com/paper/2305.13341/) (2023)
- [WeatherBench Probability: A benchmark dataset for probabilistic medium-range weather forecasting along with deep learning baseline models](https://mlsanity.com/paper/2205.00865/) (2022)

## Built on by (most cited first, 12 of 12)

- [Can AI weather models predict out-of-distribution gray swan tropical cyclones?](https://mlsanity.com/paper/2410.14932/) (2024)
- [Validating Deep Learning Weather Forecast Models on Recent High-Impact Extreme Events](https://mlsanity.com/paper/2404.17652/) (2024)
- [Numerical models outperform AI weather forecasts of record-breaking extremes](https://mlsanity.com/paper/2508.15724/) (2025)
- [Atmospheric Predictability Beyond 30 Days with Machine Learning](https://mlsanity.com/paper/2504.20238/) (2025)
- [FastNet: Improving the physical consistency of machine-learning weather prediction models through loss function design](https://mlsanity.com/paper/2509.17601/) (2025)
- [MAUSAM: An Observations-focused assessment of Global AI Weather Prediction Models During the South Asian Monsoon](https://mlsanity.com/paper/2509.01879/) (2025)
- [Lightning-Fast Convective Outlooks: Predicting Severe Convective Environments with Global AI-based Weather Models](https://mlsanity.com/paper/2406.09474/) (2024)
- [Locally Adaptive Conformal Inference for Operator Models](https://mlsanity.com/paper/2507.20975/) (2025)
- [Data-driven Mesoscale Weather Forecasting Combining Swin-Unet and Diffusion Models](https://mlsanity.com/paper/2503.19354/) (2025)
- [Skillful Subseasonal-to-Seasonal Forecasting of Extreme Events with a Multi-Sphere Coupled Probabilistic Model](https://mlsanity.com/paper/2512.12545/) (2025)
- [Do AI weather models miss extremes?](https://mlsanity.com/paper/2608.09972/) (2026)
- [Too Rare to Learn: Prescribed Cyclone Tracks Degrade a Bay of Bengal Ocean Emulator](https://mlsanity.com/paper/2609.04635/) (2026)

## Abstract (arXiv)

There has been huge recent interest in the potential of making operational weather forecasts using machine learning techniques. As they become a part of the weather forecasting toolbox, there is a pressing need to understand how well current machine learning models can simulate high-impact weather events. We compare forecasts of Storm Ciar'an, a European windstorm that caused sixteen deaths and extensive damage in Northern Europe, made by machine learning and numerical weather prediction models. The four machine learning models considered (FourCastNet, Pangu-Weather, GraphCast and FourCastNet-v2) produce forecasts that accurately capture the synoptic-scale structure of the cyclone including the position of the cloud head, shape of the warm sector and location of warm conveyor belt jet, and the large-scale dynamical drivers important for the rapid storm development such as the position of the storm relative to the upper-level jet exit. However, their ability to resolve the more detailed structures important for issuing weather warnings is more mixed. All of the machine learning models underestimate the peak amplitude of winds associated with the storm, only some machine learning models resolve the warm core seclusion and none of the machine learning models capture the sharp bent-back warm frontal gradient. Our study shows there is a great deal about the performance and properties of machine learning weather forecasts that can be derived from case studies of high-impact weather events such as Storm Ciar'an.