# Why Do Transformers Fail to Forecast Time Series In-Context?

arXiv: https://arxiv.org/abs/2510.09776 · HTML: https://mlsanity.com/paper/2510.09776/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Oct 2025.

## Built on by (most cited first, 5 of 5)

- [Universal Redundancies in Time Series Foundation Models](https://mlsanity.com/paper/2602.01605/) (2026)
- [LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications](https://mlsanity.com/paper/2608.23058/) (2026)
- [Deep Learning for Financial Time Series: A Large-Scale Benchmark of Risk-Adjusted Performance](https://mlsanity.com/paper/2603.01820/) (2026)
- [AutoAdapt: An Automated Domain Adaptation Framework for LLMs](https://mlsanity.com/paper/2603.08181/) (2026)
- [Exploring the Potential of Probabilistic Transformer for Time Series Modeling: A Report on the ST-PT Framework](https://mlsanity.com/paper/2604.26762/) (2026)

## Abstract (arXiv)

Time series forecasting (TSF) remains a challenging and largely unsolved problem in machine learning, despite significant recent efforts leveraging Large Language Models (LLMs), which predominantly rely on Transformer architectures. Empirical evidence consistently shows that even powerful Transformers often fail to outperform much simpler models, e.g., linear models, on TSF tasks; however, a rigorous theoretical understanding of this phenomenon remains limited. In this paper, we provide a theoretical analysis of Transformers' limitations for TSF through the lens of In-Context Learning (ICL) theory. Specifically, under AR(p) data, we establish that: (1) Linear Self-Attention (LSA) models textitcannot achieve lower expected MSE than classical linear models for in-context forecasting; (2) as the context length approaches to infinity, LSA asymptotically recovers the optimal linear predictor; and (3) under Chain-of-Thought (CoT) style inference, predictions collapse to the mean exponentially. We empirically validate these findings through carefully designed experiments. Our theory not only sheds light on several previously underexplored phenomena but also offers practical insights for designing more effective forecasting architectures. We hope our work encourages the broader research community to revisit the fundamental theoretical limitations of TSF and to critically evaluate the direct application of increasingly sophisticated architectures without deeper scrutiny.