# A Survey of Deep Learning and Foundation Models for Time Series Forecasting

arXiv: https://arxiv.org/abs/2401.13912 · HTML: https://mlsanity.com/paper/2401.13912/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 67 of its 141 references that are in the map.
- First posted on arXiv 25 Jan 2024.

## Builds on (most cited first, 30 of 67)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)

## Built on by (most cited first, 30 of 33)

- [Zero-shot forecasting of chaotic systems](https://mlsanity.com/paper/2409.15771/) (2024)
- [Empowering Time Series Analysis with Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2405.02358/) (2024)
- [Foundation Models for Time Series: A Survey](https://mlsanity.com/paper/2504.04011/) (2025)
- [Context parroting: A simple but tough-to-beat baseline for foundation models in scientific machine learning](https://mlsanity.com/paper/2505.11349/) (2025)
- [Deep Learning and Foundation Models for Weather Prediction: A Survey](https://mlsanity.com/paper/2501.06907/) (2025)
- [Lightweight Online Adaption for Time Series Foundation Model Forecasts](https://mlsanity.com/paper/2502.12920/) (2025)
- [Learning Interpretable Hierarchical Dynamical Systems Models from Time Series Data](https://mlsanity.com/paper/2410.04814/) (2024)
- [Empowering Time Series Forecasting with LLM-Agents](https://mlsanity.com/paper/2508.04231/) (2025)
- [Learning the Context of Errors: Black-Box Online Adaptation of Time Series Foundation Models](https://mlsanity.com/paper/2606.14222/) (2026)
- [ms-Mamba: Multi-scale Mamba for Time-Series Forecasting](https://mlsanity.com/paper/2504.07654/) (2025)
- [CondTSF: One-line Plugin of Dataset Condensation for Time Series Forecasting](https://mlsanity.com/paper/2406.02131/) (2024)
- [Flow-based Conformal Prediction for Multi-dimensional Time Series](https://mlsanity.com/paper/2502.05709/) (2025)
- [Retrieval Augmented Time Series Forecasting](https://mlsanity.com/paper/2505.04163/) (2025)
- [RDIT: Residual-based Diffusion Implicit Models for Probabilistic Time Series Forecasting](https://mlsanity.com/paper/2509.02341/) (2025)
- [Foundation Models for Structural Health Monitoring](https://mlsanity.com/paper/2404.02944/) (2024)
- [Continual Low-Rank Scaled Dot-product Attention](https://mlsanity.com/paper/2412.03214/) (2024)
- [EMForecaster: A Deep Learning Framework for Time Series Forecasting in Wireless Networks with Distribution-Free Uncertainty Quantification](https://mlsanity.com/paper/2504.00120/) (2025)
- [Adapting Time Series Foundation Models through Data Mixtures](https://mlsanity.com/paper/2603.02840/) (2026)
- [Opportunities and challenges in the application of large artificial intelligence models in radiology](https://mlsanity.com/paper/2403.16112/) (2024)
- [ARIES: Relation Assessment and Model Recommendation for Deep Time Series Forecasting](https://mlsanity.com/paper/2509.06060/) (2025)
- [Retrieval-Augmented Generation with Covariate Time Series](https://mlsanity.com/paper/2603.04951/) (2026)
- [Rethinking Adam for Time Series Forecasting: A Simple Heuristic to Improve Optimization under Distribution Shifts](https://mlsanity.com/paper/2603.10095/) (2026)
- [Towards Unified and Data-Efficient Prognostics and Health Management with Tabular Foundation Models](https://mlsanity.com/paper/2606.05481/) (2026)
- [Koopman operator theory: fundamentals, control, and applications](https://mlsanity.com/paper/2607.01819/) (2026)
- [Evaluating Time Series Foundation Models on Noisy Periodic Time Series](https://mlsanity.com/paper/2501.00889/) (2025)
- [Transformer Modeling for Both Scalability and Performance in Multivariate Time Series](https://mlsanity.com/paper/2509.19471/) (2025)
- [Frequency-aware decomposition learning for sensorless wrench estimation in vibration-rich robotic contact](https://mlsanity.com/paper/2604.12905/) (2026)
- [Forecasting Source Stability in Scientific Experiments using Temporal Learning Models: A Case Study from Tritium Monitoring](https://mlsanity.com/paper/2605.08140/) (2026)
- [DeepL'evy: Learning Heavy-Tailed Uncertainty in Highly Volatile Time Series](https://mlsanity.com/paper/2605.10364/) (2026)
- [Accurate, Efficient, and Explainable Deep Learning Approaches for Environmental Science Problems](https://mlsanity.com/paper/2605.19366/) (2026)

## Abstract (arXiv)

Deep Learning has been successfully applied to many application domains, yet its advantages have been slow to emerge for time series forecasting. For example, in the well-known Makridakis (M) Competitions, hybrids of traditional statistical or machine learning techniques have only recently become the top performers. With the recent architectural advances in deep learning being applied to time series forecasting (e.g., encoder-decoders with attention, transformers, and graph neural networks), deep learning has begun to show significant advantages. Still, in the area of pandemic prediction, there remain challenges for deep learning models: the time series is not long enough for effective training, unawareness of accumulated scientific knowledge, and interpretability of the model. To this end, the development of foundation models (large deep learning models with extensive pre-training) allows models to understand patterns and acquire knowledge that can be applied to new related problems before extensive training data becomes available. Furthermore, there is a vast amount of knowledge available that deep learning models can tap into, including Knowledge Graphs and Large Language Models fine-tuned with scientific domain knowledge. There is ongoing research examining how to utilize or inject such knowledge into deep learning models. In this survey, several state-of-the-art modeling techniques are reviewed, and suggestions for further work are provided.