# Dynamical errors in machine learning forecasts

arXiv: https://arxiv.org/abs/2504.11074 · HTML: https://mlsanity.com/paper/2504.11074/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 36 references that are in the map.
- First posted on arXiv 15 Apr 2025.

## Builds on (most cited first, 6 of 6)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [ACE2: Accurately learning subseasonal to decadal atmospheric variability and forced responses](https://mlsanity.com/paper/2411.11268/) (2024)
- [ORBIT: Oak Ridge Base Foundation Model for Earth System Predictability](https://mlsanity.com/paper/2404.14712/) (2024)
- [Spatio-temporal Dynamical Indices for Complex Systems](https://mlsanity.com/paper/2412.10069/) (2024)
- [Industry-Relevant Implicit Large-Eddy Simulation of a High-Performance Road Car via Spectral/hp Element Methods](https://mlsanity.com/paper/2009.10178/) (2020)

## Built on by (most cited first, 6 of 6)

- [FourCastNet 3: A geometric approach to probabilistic machine-learning weather forecasting at scale](https://mlsanity.com/paper/2507.12144/) (2025)
- [Hierarchy of extreme-event predictability in turbulence revealed by machine learning](https://mlsanity.com/paper/2603.13789/) (2026)
- [Benchmarking M-LTSF: Frequency and Noise-Based Evaluation of Multivariate Long Time Series Forecasting Models](https://mlsanity.com/paper/2510.04900/) (2025)
- [Parallel BiLSTM-Transformer networks for forecasting chaotic dynamics](https://mlsanity.com/paper/2510.23685/) (2025)
- [Dynamics-Aware Weighting for Deep Learning Forecasts of Chaotic Systems](https://mlsanity.com/paper/2608.22277/) (2026)
- [Do Neural PDE Solvers Learn the Right Dynamics?](https://mlsanity.com/paper/2610.06952/) (2026)

## Abstract (arXiv)

In machine learning forecasting, standard error metrics such as mean absolute error (MAE) and mean squared error (MSE) quantify discrepancies between predictions and target values. However, these metrics do not directly evaluate the physical and/or dynamical consistency of forecasts, an increasingly critical concern in scientific and engineering applications. Indeed, a fundamental yet often overlooked question is whether machine learning forecasts preserve the dynamical behavior of the underlying system. Addressing this issue is essential for assessing the fidelity of machine learning models and identifying potential failure modes, particularly in applications where maintaining correct dynamical behavior is crucial. In this work, we investigate the relationship between standard forecasting error metrics, such as MAE and MSE, and the dynamical properties of the underlying system. To achieve this goal, we use two recently developed dynamical indices: the instantaneous dimension (d), and the inverse persistence (θ). Our results indicate that larger forecast errors -- e.g., higher MSE -- tend to occur in states with higher d (higher complexity) and higher θ (lower persistence). To further assess dynamical consistency, we propose error metrics based on the dynamical indices that measure the discrepancy of the forecasted d and θ versus their correct values. Leveraging these dynamical indices-based metrics, we analyze direct and recursive forecasting strategies for three canonical datasets -- Lorenz, Kuramoto-Sivashinsky equation, and Kolmogorov flow -- as well as a real-world weather forecasting task. Our findings reveal substantial distortions in dynamical properties in ML forecasts, especially for long forecast lead times or long recursive simulations, providing complementary information on ML forecast fidelity that can be used to improve ML models.