mlsanityData to 7 Oct 2026

Timer-XL: Long-Context Transformers for Unified Time Series Forecasting

arXiv 2410.04803 ↗First posted 7 Oct 2024cs.LG stat.MLcited by 83

Not on the map. Why?

Abstract (arXiv)

We present Timer-XL, a causal Transformer for unified time series forecasting. To uniformly predict multidimensional time series, we generalize next token prediction, predominantly adopted for 1D token sequences, to multivariate next token prediction. The paradigm formulates various forecasting tasks as a long-context prediction problem. We opt for decoder-only Transformers that capture causal dependencies from varying-length contexts for unified forecasting, making predictions on non-stationary univariate time series, multivariate series with complicated dynamics and correlations, as well as covariate-informed contexts that include exogenous variables. Technically, we propose a universal TimeAttention to capture fine-grained intra- and inter-series dependencies of flattened time series tokens (patches), which is further enhanced by deft position embedding for temporal causality and variable equivalence. Timer-XL achieves state-of-the-art performance across task-specific forecasting benchmarks through a unified approach. Based on large-scale pre-training, Timer-XL achieves state-of-the-art zero-shot performance, making it a promising architecture for pre-trained time series models. Code is available at this repository: https://github.com/thuml/Timer-XL.

Key facts. Cited by 83 papers in the map (data through 7 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

40 of 56 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Segment Anythingcited by 8,158
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2023A Survey of Large Language Modelscited by 2,537
  12. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
28 more
  1. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  2. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  3. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  4. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  5. 2017Deep Setscited by 1,394
  6. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  7. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,030
  8. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,004
  9. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 906
  10. 2023A Survey on Multimodal Large Language Modelscited by 898
  11. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 869
  12. 2019Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastingcited by 748
  13. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 744
  14. 2024Chronos: Learning the Language of Time Seriescited by 712
  15. 2023A decoder-only foundation model for time-series forecastingcited by 650
  16. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
  17. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  18. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
  19. 2022Non-stationary Transformers: Exploring the Stationarity in Time Series Forecastingcited by 385
  20. 2021SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interactioncited by 335
  21. 2021Spectral Temporal Graph Neural Network for Multivariate Time-series Forecastingcited by 298
  22. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 280
  23. 2024TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variablescited by 241
  24. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 198
  25. 2020Forecasting day-ahead electricity prices: A review of state-of-the-art algorithms, best practices and an open-access benchmarkcited by 105
  26. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 71
  27. 2023UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognitioncited by 64
  28. 2024UniTST: Effectively Modeling Inter-Series and Intra-Series Dependencies for Multivariate Time Series Forecastingcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 36; 2026: 45.

83 papers.

Sort:

By area:

  1. 2025Sundial: A Family of Highly Capable Time Series Foundation Modelscited by 168
  2. 2025MTBench: A Multimodal Time Series Benchmark for Temporal Reasoning and Question Answeringcited by 50
  3. 2025TS-RAG: Retrieval-Augmented Generation based Time Series Foundation Models are Stronger Zero-Shot Forecastercited by 36
  4. 2026Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scalingcited by 24
  5. 2025Foundation Models for Time Series: A Surveycited by 19
  6. 2024xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memoriescited by 18
  7. 2025SciTS: Scientific Time Series Understanding and Generation with LLMscited by 13
  8. 2025Diffusion Models for Time Series Forecasting: A Surveycited by 11
  9. 2025BEDTime: A Unified Benchmark for Automatically Describing Time Seriescited by 11
  10. 2026Reverso: Efficient Time Series Foundation Models for Zero-shot Forecastingcited by 11
  11. 2025TimesBERT: A BERT-Style Foundation Model for Time Series Understandingcited by 10
  12. 2026TiRex-2: Generalizing TiRex to Multivariate Data and Streamingcited by 10
38 more
  1. 2024OTIS: Learning High-Quality Time Series Features With Tiny Encoderscited by 8
  2. 2025TimeCopilotcited by 8
  3. 2025Kairos: Toward Adaptive and Parameter-Efficient Time Series Foundation Modelscited by 8
  4. 2025Learning General Causal Structures with Hidden Dynamic Process for Climate Analysiscited by 6
  5. 2025CITRAS: Covariate-Informed Transformer for Time Series Forecastingcited by 5
  6. 2025TRACE: Grounding Time Series in Context for Multimodal Embedding and Retrievalcited by 5
  7. 2025CoRA: Covariate-Aware Adaptation of Time Series Foundation Modelscited by 5
  8. 2025UniCA: Unified Covariate Adaptation for Time Series Foundation Modelcited by 4
  9. 2026Unlocking the Value of Text: Event-Driven Reasoning and Multi-Level Alignment for Time Series Forecastingcited by 4
  10. 2025Investigating Compositional Reasoning in Time Series Foundation Modelscited by 3
  11. 2025Are foundation models useful feature extractors for electroencephalography analysis?cited by 3
  12. 2026Seg-MoE: Multi-Resolution Segment-wise Mixture-of-Experts for Time Series Forecasting Transformerscited by 3
  13. 2026TimeSqueeze: Dynamic Patching for Efficient Time Series Forecastingcited by 3
  14. 2026Into the ORBIT for Time Series: Training Regimes for Foundation Modelscited by 3
  15. 2025Super-Linear: A Lightweight Pretrained Mixture of Linear Experts for Time Series Forecastingcited by 2
  16. 2026CPiRi: Channel Permutation-Invariant Relational Interaction for Multivariate Time Series Forecastingcited by 2
  17. 2026Impermanent: A Live Benchmark for Temporal Generalization in Time Series Forecastingcited by 2
  18. 2026Sonata: A Hybrid World Model for Inertial Kinematics under Clinical Data Scarcitycited by 2
  19. 2026Why Do Time Series Models Need Long Context Windows?cited by 2
  20. 2025Mitigating Data Scarcity in Time Series Analysis: A Foundation Model with Series-Symbol Data Generationcited by 1
  21. 2025Time Tracker: Mixture-of-Experts-Enhanced Foundation Time Series Forecasting Model with Decoupled Training Pipelinescited by 1
  22. 2025Beyond All-to-All: Causal-Aligned Transformer with Dynamic Structure Learning for Multivariate Time Series Forecastingcited by 1
  23. 2025Fourier Basis Mapping: A Time-Frequency Learning Framework for Time Series Forecastingcited by 1
  24. 2025HGTS-Former: Hierarchical HyperGraph Transformer for Multivariate Time Series Analysiscited by 1
  25. 2025ChaosNexus: A Foundation Model for ODE-based Chaotic System Forecasting with Hierarchical Multi-scale Awarenesscited by 1
  26. 2025Beyond Model Ranking: Predictability-Aligned Evaluation for Time Series Forecastingcited by 1
  27. 2025FreqFlow: Long-term forecasting using lightweight flow matchingcited by 1
  28. 2025RED-F: Reconstruction-Elimination based Dual-stream Contrastive Forecasting for Multivariate Time Series Anomaly Predictioncited by 1
  29. 2026Time series forecasting with Hahn Kolmogorov-Arnold networkscited by 1
  30. 2026To See Far, Look Close: Evolutionary Forecasting for Long-term Time Seriescited by 1
  31. 2026LASS-ODE: Scaling ODE Computations to Connect Foundation Models with Dynamical Physical Systemscited by 1
  32. 2026DiTS: Multimodal Diffusion Transformers Are Time Series Forecasterscited by 1
  33. 2026Large Causal Models for Temporal Causal Discoverycited by 1
  34. 2026DualWeaver: Synergistic Feature Weaving Surrogates for Multivariate Forecasting with Univariate Time Series Foundation Modelscited by 1
  35. 2026Retrieval-Augmented Generation with Covariate Time Seriescited by 1
  36. 2026Regression Models Meet Foundation Models: A Hybrid-AI Approach to Practical Electricity Price Forecastingcited by 1
  37. 2026Dynamic Tokenization via Reinforcement Patching: End-to-end Training and Zero-shot Transfercited by 1
  38. 2026Predicting Power-System Dynamic Trajectories with Foundation Modelscited by 1

50 most cited of 83 All 83 (JSON)