# Timer-XL: Long-Context Transformers for Unified Time Series Forecasting

arXiv: https://arxiv.org/abs/2410.04803 · HTML: https://mlsanity.com/paper/2410.04803/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 56 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Deep Sets](https://mlsanity.com/paper/1703.06114/) (2017)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Chronos: Learning the Language of Time Series](https://mlsanity.com/paper/2403.07815/) (2024)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)

## Built on by (most cited first, 30 of 87)

- [Sundial: A Family of Highly Capable Time Series Foundation Models](https://mlsanity.com/paper/2502.00816/) (2025)
- [MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answering](https://mlsanity.com/paper/2503.16858/) (2025)
- [TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecaster](https://mlsanity.com/paper/2503.07649/) (2025)
- [Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scaling](https://mlsanity.com/paper/2603.04791/) (2026)
- [Foundation Models for Time Series: A Survey](https://mlsanity.com/paper/2504.04011/) (2025)
- [xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memories](https://mlsanity.com/paper/2410.16928/) (2024)
- [SciTS: Scientific Time Series Understanding and Generation with LLMs](https://mlsanity.com/paper/2510.03255/) (2025)
- [Reverso: Efficient Time Series Foundation Models for Zero-shot Forecasting](https://mlsanity.com/paper/2602.17634/) (2026)
- [TiRex-2: Generalizing TiRex to Multivariate Data and Streaming](https://mlsanity.com/paper/2607.01204/) (2026)
- [BEDTime: A Unified Benchmark for Automatically Describing Time Series](https://mlsanity.com/paper/2509.05215/) (2025)
- [Diffusion Models for Time Series Forecasting: A Survey](https://mlsanity.com/paper/2507.14507/) (2025)
- [TimesBERT: A BERT-Style Foundation Model for Time Series Understanding](https://mlsanity.com/paper/2502.21245/) (2025)
- [TimeCopilot](https://mlsanity.com/paper/2509.00616/) (2025)
- [Kairos: Toward Adaptive and Parameter-Efficient Time Series Foundation Models](https://mlsanity.com/paper/2509.25826/) (2025)
- [OTIS: Learning High-Quality Time Series Features With Tiny Encoders](https://mlsanity.com/paper/2410.07299/) (2024)
- [Learning General Causal Structures with Hidden Dynamic Process for Climate Analysis](https://mlsanity.com/paper/2501.12500/) (2025)
- [TRACE: Grounding Time Series in Context for Multimodal Embedding and Retrieval](https://mlsanity.com/paper/2506.09114/) (2025)
- [UniCA: Unified Covariate Adaptation for Time Series Foundation Model](https://mlsanity.com/paper/2506.22039/) (2025)
- [CoRA: Covariate-Aware Adaptation of Time Series Foundation Models](https://mlsanity.com/paper/2510.12681/) (2025)
- [CITRAS: Covariate-Informed Transformer for Time Series Forecasting](https://mlsanity.com/paper/2503.24007/) (2025)
- [Unlocking the Value of Text: Event-Driven Reasoning and Multi-Level Alignment for Time Series Forecasting](https://mlsanity.com/paper/2603.15452/) (2026)
- [Impermanent: A Live Benchmark for Temporal Generalization in Time Series Forecasting](https://mlsanity.com/paper/2603.08707/) (2026)
- [Investigating Compositional Reasoning in Time Series Foundation Models](https://mlsanity.com/paper/2502.06037/) (2025)
- [Are foundation models useful feature extractors for electroencephalography analysis?](https://mlsanity.com/paper/2502.21086/) (2025)
- [Seg-MoE: Multi-Resolution Segment-wise Mixture-of-Experts for Time Series Forecasting Transformers](https://mlsanity.com/paper/2601.21641/) (2026)
- [TimeSqueeze: Dynamic Patching for Efficient Time Series Forecasting](https://mlsanity.com/paper/2603.11352/) (2026)
- [Why Do Time Series Models Need Long Context Windows?](https://mlsanity.com/paper/2606.01999/) (2026)
- [Into the ORBIT for Time Series: Training Regimes for Foundation Models](https://mlsanity.com/paper/2608.13262/) (2026)
- [Super-Linear: A Lightweight Pretrained Mixture of Linear Experts for Time Series Forecasting](https://mlsanity.com/paper/2509.15105/) (2025)
- [RED-F: Reconstruction-Elimination based Dual-stream Contrastive Forecasting for Multivariate Time Series Anomaly Prediction](https://mlsanity.com/paper/2511.20044/) (2025)

## Abstract (arXiv)

We present Timer-XL, a causal Transformer for unified time series forecasting. To uniformly predict multidimensional time series, we generalize next token prediction, predominantly adopted for 1D token sequences, to multivariate next token prediction. The paradigm formulates various forecasting tasks as a long-context prediction problem. We opt for decoder-only Transformers that capture causal dependencies from varying-length contexts for unified forecasting, making predictions on non-stationary univariate time series, multivariate series with complicated dynamics and correlations, as well as covariate-informed contexts that include exogenous variables. Technically, we propose a universal TimeAttention to capture fine-grained intra- and inter-series dependencies of flattened time series tokens (patches), which is further enhanced by deft position embedding for temporal causality and variable equivalence. Timer-XL achieves state-of-the-art performance across task-specific forecasting benchmarks through a unified approach. Based on large-scale pre-training, Timer-XL achieves state-of-the-art zero-shot performance, making it a promising architecture for pre-trained time series models. Code is available at this repository: https://github.com/thuml/Timer-XL.