# Temporal horizons in forecasting: a performance-learnability trade-off

arXiv: https://arxiv.org/abs/2506.03889 · HTML: https://mlsanity.com/paper/2506.03889/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 78 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Better & Faster Large Language Models via Multi-token Prediction](https://mlsanity.com/paper/2404.19737/) (2024)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [On Neural Differential Equations](https://mlsanity.com/paper/2202.02435/) (2022)
- [PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers](https://mlsanity.com/paper/2308.05732/) (2023)
- [FourCastNet: Accelerating Global High-Resolution Weather Forecasting using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2208.05419/) (2022)
- [Forecasting Sequential Data using Consistent Koopman Autoencoders](https://mlsanity.com/paper/2003.02236/) (2020)
- [AntisymmetricRNN: A Dynamical System View on Recurrent Neural Networks](https://mlsanity.com/paper/1902.09689/) (2019)
- [A Deeper Look at Experience Replay](https://mlsanity.com/paper/1712.01275/) (2017)
- [Lipschitz Recurrent Neural Networks](https://mlsanity.com/paper/2006.12070/) (2020)
- [Physics-informed Autoencoders for Lyapunov-stable Fluid Flow Prediction](https://mlsanity.com/paper/1905.10866/) (2019)
- [Learning Compositional Koopman Operators for Model-Based Control](https://mlsanity.com/paper/1910.08264/) (2019)
- [On the difficulty of learning chaotic dynamics with RNNs](https://mlsanity.com/paper/2110.07238/) (2021)
- [Were RNNs All We Needed?](https://mlsanity.com/paper/2410.01201/) (2024)
- [Learning to Plan for Language Modeling from Unlabeled Data](https://mlsanity.com/paper/2404.00614/) (2024)
- [Motion Generation Using Bilateral Control-Based Imitation Learning with Autoregressive Learning](https://mlsanity.com/paper/2011.06192/) (2020)
- [Stepping on the Edge: Curvature Aware Learning Rate Tuners](https://mlsanity.com/paper/2407.06183/) (2024)
- [Improved memory in recurrent neural networks with sequential non-normal dynamics](https://mlsanity.com/paper/1905.13715/) (2019)
- [Preferential Temporal Difference Learning](https://mlsanity.com/paper/2106.06508/) (2021)
- [A Modified Multiple Shooting Algorithm for Parameter Estimation in ODEs Using Adjoint Sensitivity Analysis](https://mlsanity.com/paper/2006.15740/) (2020)
- [A loss discounting framework for model averaging and selection in time series models](https://mlsanity.com/paper/2201.12045/) (2022)

## Built on by (most cited first, 1 of 1)

- [Large Distant Gradients Need Not Be Reliable: reliability-weighted credit assignment for long-horizon autoregressive forecasting](https://mlsanity.com/paper/2609.12890/) (2026)

## Abstract (arXiv)

When training autoregressive models to forecast dynamical systems, a critical question arises: how far into the future should the model be trained to predict for optimal performance? In this work, we address this question by analyzing the relationship between the geometry of the loss landscape and the training time horizon. Using dynamical systems theory, we prove that loss minima for long horizons generalize well to short-term forecasts, whereas minima found on short horizons result in worse long-term predictions. However, we also prove that the loss landscape becomes rougher as the training horizon grows, making long-horizon training inherently challenging. We validate our theory through numerical experiments and discuss practical implications for selecting training horizons. Our results provide a principled foundation for hyperparameter optimization in autoregressive forecasting models.