# Foundation Models for Time Series: A Survey

arXiv: https://arxiv.org/abs/2504.04011 · HTML: https://mlsanity.com/paper/2504.04011/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 64 of its 133 references that are in the map.
- First posted on arXiv 5 Apr 2025.

## Builds on (most cited first, 30 of 64)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [Time Series Forecasting With Deep Learning: A Survey](https://mlsanity.com/paper/2004.13408/) (2020)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy](https://mlsanity.com/paper/2110.02642/) (2021)
- [Time-Series Representation Learning via Temporal and Contextual Contrasting](https://mlsanity.com/paper/2106.14112/) (2021)
- [Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts](https://mlsanity.com/paper/2409.16040/) (2024)

## Built on by (most cited first, 23 of 23)

- [Reverso: Efficient Time Series Foundation Models for Zero-shot Forecasting](https://mlsanity.com/paper/2602.17634/) (2026)
- [Falcon-X: A Time Series Foundation Model for Heterogeneous Multivariate Modeling](https://mlsanity.com/paper/2605.27286/) (2026)
- [Explainable Load Forecasting with Covariate-Informed Time Series Foundation Models](https://mlsanity.com/paper/2604.28149/) (2026)
- [Causal Data Augmentation for Robust Fine-Tuning of Tabular Foundation Models](https://mlsanity.com/paper/2601.04110/) (2026)
- [Diversified Scaling Inference in Time Series Foundation Models](https://mlsanity.com/paper/2601.17376/) (2026)
- [Why Do Time Series Models Need Long Context Windows?](https://mlsanity.com/paper/2606.01999/) (2026)
- [Into the ORBIT for Time Series: Training Regimes for Foundation Models](https://mlsanity.com/paper/2608.13262/) (2026)
- [XLinear: A Lightweight and Accurate MLP-Based Model for Long-Term Time Series Forecasting with Exogenous Inputs](https://mlsanity.com/paper/2601.09237/) (2026)
- [TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Models](https://mlsanity.com/paper/2606.06285/) (2026)
- [WPBench: A Comprehensive Benchmark for Wind Power Forecasting](https://mlsanity.com/paper/2609.24444/) (2026)
- [VTBench: A Multimodal Framework for Time-Series Classification with Chart-Based Representations](https://mlsanity.com/paper/2604.27259/) (2026)
- [Heterogeneous Scientific Foundation Model Collaboration](https://mlsanity.com/paper/2604.27351/) (2026)
- [Towards Resilient and Autonomous Networks: A BlueSky Vision on AI-Native 6G](https://mlsanity.com/paper/2605.21395/) (2026)
- [Ground-Truth Neighborhood Regularization for Reinforcement Learning Post-Training of Time Series Foundation Models](https://mlsanity.com/paper/2608.08010/) (2026)
- [Comparative Analysis of Time Series Foundation Models for Demographic Forecasting: Enhancing Predictive Accuracy in US Population Dynamics](https://mlsanity.com/paper/2508.11680/) (2025)
- [EngineAD: A Real-World Vehicle Engine Anomaly Detection Dataset](https://mlsanity.com/paper/2603.25955/) (2026)
- [Temporal Feature Extractors in EEG Foundation Models: A Controlled Comparison Including a Pretrained Time-Series Model](https://mlsanity.com/paper/2606.30104/) (2026)
- [Exploring Zero-Shot Foundation Models for Multivariate Time Series Anomaly Detection](https://mlsanity.com/paper/2607.12454/) (2026)
- [Learning to Forecast Crop Growth from Earth Observation Data](https://mlsanity.com/paper/2608.14281/) (2026)
- [Involving before Evolving: A Vision for Trustworthy Enterprise Digital Twin Engineering](https://mlsanity.com/paper/2609.13071/) (2026)
- [A Unified Scaling Law for Time Series Foundation Models](https://mlsanity.com/paper/2610.05269/) (2026)
- [Inverse Synthesis of Shock Time Series from Shock Response Spectra Using Ensembles of Conditional Variational Autoencoders](https://mlsanity.com/paper/2610.12820/) (2026)
- [Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting](https://mlsanity.com/paper/2610.19603/) (2026)

## Abstract (arXiv)

Transformer-based foundation models have emerged as a dominant paradigm in time series analysis, offering unprecedented capabilities in tasks such as forecasting, anomaly detection, classification, trend analysis and many more time series analytical tasks. This survey provides a comprehensive overview of the current state of the art pre-trained foundation models, introducing a novel taxonomy to categorize them across several dimensions. Specifically, we classify models by their architecture design, distinguishing between those leveraging patch-based representations and those operating directly on raw sequences. The taxonomy further includes whether the models provide probabilistic or deterministic predictions, and whether they are designed to work with univariate time series or can handle multivariate time series out of the box. Additionally, the taxonomy encompasses model scale and complexity, highlighting differences between lightweight architectures and large-scale foundation models. A unique aspect of this survey is its categorization by the type of objective function employed during training phase. By synthesizing these perspectives, this survey serves as a resource for researchers and practitioners, providing insights into current trends and identifying promising directions for future research in transformer-based time series modeling.