# Why Cannot Neural Networks Master Extrapolation? Insights from Physical Laws

arXiv: https://arxiv.org/abs/2510.04102 · HTML: https://mlsanity.com/paper/2510.04102/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 16 references that are in the map.
- First posted on arXiv 5 Oct 2025.

## Builds on (most cited first, 10 of 10)

- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [AI Feynman: a Physics-Inspired Method for Symbolic Regression](https://mlsanity.com/paper/1905.11481/) (2019)
- [Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Experts](https://mlsanity.com/paper/2410.10469/) (2024)
- [TSFM-Bench: A Comprehensive and Unified Benchmark of Foundation Models for Time Series Forecasting](https://mlsanity.com/paper/2410.11802/) (2024)
- [Estimating Generalization under Distribution Shifts via Domain-Invariant Representations](https://mlsanity.com/paper/2007.03511/) (2020)
- [SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention](https://mlsanity.com/paper/2402.10198/) (2024)
- [Performance of Zero-Shot Time Series Foundation Models on Cloud Data](https://mlsanity.com/paper/2502.12944/) (2025)
- [Robust Generalization despite Distribution Shift via Minimum Discriminating Information](https://mlsanity.com/paper/2106.04443/) (2021)

## Built on by (most cited first, 2 of 2)

- [A Special Case of Quadratic Extrapolation Under the Neural Tangent Kernel](https://mlsanity.com/paper/2512.15749/) (2025)
- [Routing Dense Layouts with History-Aware Offline Reinforcement Learning using LSTM](https://mlsanity.com/paper/2609.08232/) (2026)

## Abstract (arXiv)

Motivated by the remarkable success of Foundation Models (FMs) in language modeling, there has been growing interest in developing FMs for time series prediction, given the transformative power such models hold for science and engineering. This culminated in significant success of FMs in short-range forecasting settings. However, extrapolation or long-range forecasting remains elusive for FMs, which struggle to outperform even simple baselines. This contrasts with physical laws which have strong extrapolation properties, and raises the question of the fundamental difference between the structure of neural networks and physical laws. In this work, we identify and formalize a fundamental property characterizing the ability of statistical learning models to predict more accurately outside of their training domain, hence explaining performance deterioration for deep learning models in extrapolation settings. In addition to a theoretical analysis, we present empirical results showcasing the implications of this property on current deep learning architectures. Our results not only clarify the root causes of the extrapolation gap but also suggest directions for designing next-generation forecasting models capable of mastering extrapolation.