# Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting

arXiv: https://arxiv.org/abs/2310.08278 · HTML: https://mlsanity.com/paper/2310.08278/

## Key facts

- Cited by 113 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 66 references that are in the map.
- First posted on arXiv 12 Oct 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting](https://mlsanity.com/paper/2205.14415/) (2022)
- [Data Distributional Properties Drive Emergent In-Context Learning in Transformers](https://mlsanity.com/paper/2205.05055/) (2022)
- [Monash Time Series Forecasting Archive](https://mlsanity.com/paper/2105.06643/) (2021)
- [A Multi-Horizon Quantile Recurrent Forecaster](https://mlsanity.com/paper/1711.11053/) (2017)
- [TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting](https://mlsanity.com/paper/2310.04948/) (2023)
- [High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes](https://mlsanity.com/paper/1910.03002/) (2019)
- [ETSformer: Exponential Smoothing Transformers for Time-series Forecasting](https://mlsanity.com/paper/2202.01381/) (2022)
- [Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows](https://mlsanity.com/paper/2002.06103/) (2020)
- [CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting](https://mlsanity.com/paper/2202.01575/) (2022)
- [Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case](https://mlsanity.com/paper/2001.08317/) (2020)
- [Deep Learning for Time Series Forecasting: Tutorial and Literature Survey](https://mlsanity.com/paper/2004.10240/) (2020)
- [Broken Neural Scaling Laws](https://mlsanity.com/paper/2210.14891/) (2022)
- [Ti-MAE: Self-Supervised Masked Time Series Autoencoders](https://mlsanity.com/paper/2301.08871/) (2023)

## Built on by (most cited first, 30 of 113)

- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [ForecastPFN: Synthetically-Trained Zero-Shot Forecasting](https://mlsanity.com/paper/2311.01933/) (2023)
- [Moirai 2.0: When Less Is More for Time Series Forecasting](https://mlsanity.com/paper/2511.11698/) (2025)
- [Retrieval Augmented Time Series Forecasting](https://mlsanity.com/paper/2411.08249/) (2024)
- [BlackMamba: Mixture of Experts for State-Space Models](https://mlsanity.com/paper/2402.01771/) (2024)
- [GATGPT: A Pre-trained Large Language Model with Graph Attention Network for Spatiotemporal Imputation](https://mlsanity.com/paper/2311.14332/) (2023)
- [A Survey of Deep Learning and Foundation Models for Time Series Forecasting](https://mlsanity.com/paper/2401.13912/) (2024)
- [Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization](https://mlsanity.com/paper/2412.05244/) (2024)
- [MIRA: Medical Time Series Foundation Model for Real-World Health Data](https://mlsanity.com/paper/2506.07584/) (2025)
- [TimeSeriesScientist: A General-Purpose AI Agent for Time Series Analysis](https://mlsanity.com/paper/2510.01538/) (2025)
- [TimeRAF: Retrieval-Augmented Foundation model for Zero-shot Time Series Forecasting](https://mlsanity.com/paper/2412.20810/) (2024)
- [Learning Interpretable Hierarchical Dynamical Systems Models from Time Series Data](https://mlsanity.com/paper/2410.04814/) (2024)
- [Pre-trained Forecasting Models: Strong Zero-Shot Feature Extractors for Time Series Classification](https://mlsanity.com/paper/2510.26777/) (2025)
- [Learning the Context of Errors: Black-Box Online Adaptation of Time Series Foundation Models](https://mlsanity.com/paper/2606.14222/) (2026)
- [Macroeconomic Forecasting with Large Language Models](https://mlsanity.com/paper/2407.00890/) (2024)
- [Efficient Time Series Processing for Transformers and State-Space Models through Token Merging](https://mlsanity.com/paper/2405.17951/) (2024)
- [MoTime: A Dataset Suite for Multimodal Time Series Forecasting](https://mlsanity.com/paper/2505.15072/) (2025)
- [GlucoFM: A Dual-Stream Foundation Model for Continuous Glucose Monitoring](https://mlsanity.com/paper/2605.30865/) (2026)
- [Unlocking Pretrained Vision Transformers for Time Series Classification](https://mlsanity.com/paper/2506.08641/) (2025)
- [Foundation models for time series forecasting: Application in conformal prediction](https://mlsanity.com/paper/2507.08858/) (2025)
- [Nexus : An Agentic Framework for Time Series Forecasting](https://mlsanity.com/paper/2605.14389/) (2026)
- [Towards a General Intelligence and Interface for Wearable Health Data](https://mlsanity.com/paper/2605.22759/) (2026)
- [Towards Long-Context Time Series Foundation Models](https://mlsanity.com/paper/2409.13530/) (2024)
- [LAST SToP For Modeling Asynchronous Time Series](https://mlsanity.com/paper/2502.01922/) (2025)
- [Byte Pair Encoding for Efficient Time Series Forecasting](https://mlsanity.com/paper/2505.14411/) (2025)
- [TempusBench: An Evaluation Framework for Time-Series Forecasting](https://mlsanity.com/paper/2604.11529/) (2026)
- [Adaptive Conformal Anomaly Detection with Time Series Foundation Models for Signal Monitoring](https://mlsanity.com/paper/2604.20122/) (2026)
- [The impact of data set similarity and diversity on transfer learning success in time series forecasting](https://mlsanity.com/paper/2404.06198/) (2024)
- [Survey and Taxonomy: The Role of Data-Centric AI in Transformer-Based Time Series Forecasting](https://mlsanity.com/paper/2407.19784/) (2024)
- [Strada-LLM: Graph LLM for traffic prediction](https://mlsanity.com/paper/2410.20856/) (2024)

## Abstract (arXiv)

Over the past years, foundation models have caused a paradigm shift in machine learning due to their unprecedented capabilities for zero-shot and few-shot generalization. However, despite the success of foundation models in modalities such as natural language processing and computer vision, the development of foundation models for time series forecasting has lagged behind. We present Lag-Llama, a general-purpose foundation model for univariate probabilistic time series forecasting based on a decoder-only transformer architecture that uses lags as covariates. Lag-Llama is pretrained on a large corpus of diverse time series data from several domains, and demonstrates strong zero-shot generalization capabilities compared to a wide range of forecasting models on downstream datasets across domains. Moreover, when fine-tuned on relatively small fractions of such previously unseen datasets, Lag-Llama achieves state-of-the-art performance, outperforming prior deep learning approaches, emerging as the best general-purpose model on average. Lag-Llama serves as a strong contender to the current state-of-art in time series forecasting and paves the way for future advancements in foundation models tailored to time series data.