# PhysMetrics.Weather: An Evaluation Framework for Physical Consistency in ML Weather Models

arXiv: https://arxiv.org/abs/2606.10642 · HTML: https://mlsanity.com/paper/2606.10642/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 39 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Builds on (most cited first, 13 of 13)

- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [GraphCast: Learning skillful medium-range global weather forecasting](https://mlsanity.com/paper/2212.12794/) (2022)
- [Neural General Circulation Models for Weather and Climate](https://mlsanity.com/paper/2311.07222/) (2023)
- [WeatherBench 2: A benchmark for the next generation of data-driven global weather models](https://mlsanity.com/paper/2308.15560/) (2023)
- [GenCast: Diffusion-based ensemble forecasting for medium-range weather](https://mlsanity.com/paper/2312.15796/) (2023)
- [Pangu-Weather: A 3D High-Resolution Model for Fast and Accurate Global Weather Forecast](https://mlsanity.com/paper/2211.02556/) (2022)
- [ClimODE: Climate and Weather Forecasting with Physics-informed Neural ODEs](https://mlsanity.com/paper/2404.10024/) (2024)
- [ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction](https://mlsanity.com/paper/2402.00712/) (2024)
- [Fixing the Double Penalty in Data-Driven Weather Forecasting Through a Modified Spherical Harmonic Loss Function](https://mlsanity.com/paper/2501.19374/) (2025)
- [FuXi Weather: A data-to-forecast machine learning system for global weather](https://mlsanity.com/paper/2408.05472/) (2024)
- [FastNet: Improving the physical consistency of machine-learning weather prediction models through loss function design](https://mlsanity.com/paper/2509.17601/) (2025)
- [Assessing the Geographic Generalization and Physical Consistency of Generative Models for Climate Downscaling](https://mlsanity.com/paper/2510.13722/) (2025)

## Built on by (most cited first, 5 of 5)

- [Toward Mechanistic Interpretability of an AI Foundation Model Fine-Tuned for Atmospheric Chemistry](https://mlsanity.com/paper/2607.20778/) (2026)
- [Stochastically Perturbed Weights: Ensembles from Deterministic Machine-Learning Weather Models](https://mlsanity.com/paper/2609.08412/) (2026)
- [Every Fixed Metric Has a Blind Spot: A Learned Atmospheric Critic for Scoring Forecast Realism](https://mlsanity.com/paper/2609.18381/) (2026)
- [Evaluating Dynamical Fidelity through Predictive Structure in Physical Representations](https://mlsanity.com/paper/2609.34627/) (2026)
- [Strategic Governance of AI Models in Earth Science](https://mlsanity.com/paper/2610.10560/) (2026)

## Abstract (arXiv)

Machine learning weather prediction (MLWP) models have achieved impressive forecasting performance at a small fraction of the computational costs required for traditional physics-based methods. However, they are primarily (1) data-driven and (2) evaluated using pixel-wide error metrics (e.g., RMSE), so there are no guarantees that their forecasts are consistent with known physical laws. We introduce PhysMetrics.Weather, an evaluation framework that assesses the physical realism of MLWP models across three types of metrics: conservation, spectral, and dynamical. By quantifying physical realism, this tool guides the development of physics-informed architectures and helps evaluate whether MLWP models are reliable for operational use. Our framework is available on Github at https://github.com/Emmakast/PhysMetrics.Weather.