mlsanityData to 14 Oct 2026

Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting

arXiv 2106.13008 ↗First posted 24 Jun 2021cs.LG cs.AIcited by 1,676

Not on the map. Why?

Cited by 40 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Extending the forecasting time is a critical demand for real applications, such as extreme weather early warning and long-term energy consumption planning. This paper studies the long-term forecasting problem of time series. Prior Transformer-based models adopt various self-attention mechanisms to discover the long-range dependencies. However, intricate temporal patterns of the long-term future prohibit the model from finding reliable dependencies. Also, Transformers have to adopt the sparse versions of point-wise self-attentions for long series efficiency, resulting in the information utilization bottleneck. Going beyond Transformers, we design Autoformer as a novel decomposition architecture with an Auto-Correlation mechanism. We break with the pre-processing convention of series decomposition and renovate it as a basic inner block of deep models. This design empowers Autoformer with progressive decomposition capacities for complex time series. Further, inspired by the stochastic process theory, we design the Auto-Correlation mechanism based on the series periodicity, which conducts the dependencies discovery and representation aggregation at the sub-series level. Auto-Correlation outperforms self-attention in both efficiency and accuracy. In long-term forecasting, Autoformer yields state-of-the-art accuracy, with a 38% relative improvement on six benchmarks, covering five practical applications: energy, traffic, economics, weather and disease. Code is available at this repository: urlhttps://github.com/thuml/Autoformer.

Key facts. Cited by 1,676 papers in the map (data through 14 Oct 2026). Builds on 26 of its 55 references that are in the map. First posted on arXiv 24 Jun 2021.

Builds on

26 of 55 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  10. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  11. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  12. 2020Reformer: The Efficient Transformercited by 1,668
14 more
  1. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
  2. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,013
  3. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 879
  4. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 753
  5. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 607
  6. 2017A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Predictioncited by 245
  7. 2017A Multi-Horizon Quantile Recurrent Forecastercited by 211
  8. 2019Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecastingcited by 161
  9. 2018Temporal Pattern Attention for Multivariate Time Series Forecastingcited by 125
  10. 2017Conditional Time Series Forecasting with Convolutional Neural Networkscited by 123
  11. 2017Attend and Diagnose: Clinical Time Series Analysis using Attention Modelscited by 105
  12. 2021Data-driven Prediction of General Hamiltonian Dynamics via Learning Exactly-Symplectic Mapscited by 41
  13. 2017Long-term Forecasting using Higher Order Tensor RNNscited by 34
  14. 2018Deep Factors with Gaussian Processes for Forecastingcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 12; 2022: 66; 2023: 185; 2024: 403; 2025: 521; 2026: 489.

1,676 papers.

Sort:

By area:

  1. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  2. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
  3. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
  4. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,058
  5. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 931
  6. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  7. 2023A decoder-only foundation model for time-series forecastingcited by 692
  8. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 632
  9. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 614
  10. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 500
  11. 2022Transformers in Time Series: A Surveycited by 497
  12. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 482
38 more
  1. 2021Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancycited by 394
  2. 2022Non-stationary Transformers: Exploring the Stationarity in Time Series Forecastingcited by 391
  3. 2024TimeMixer: Decomposable Multiscale Mixing for Time Series Forecastingcited by 346
  4. 2023Long-term Forecasting with TiDE: Time-series Dense Encodercited by 332
  5. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 287
  6. 2024TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methodscited by 249
  7. 2024TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variablescited by 248
  8. 2023TSMixer: An All-MLP Architecture for Time Series Forecastingcited by 240
  9. 2024GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluationcited by 212
  10. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 208
  11. 2022Diffusion-based Time Series Imputation and Forecasting with Structured State Space Modelscited by 198
  12. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 197
  13. 2023FITS: Modeling Time Series with 10k Parameterscited by 195
  14. 2025Sundial: A Family of Highly Capable Time Series Foundation Modelscited by 180
  15. 2023TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecastingcited by 174
  16. 2022PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecastingcited by 172
  17. 2023Frequency-domain MLPs are More Effective Learners in Time Series Forecastingcited by 172
  18. 2023TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Seriescited by 169
  19. 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 168
  20. 2022Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structurescited by 160
  21. 2023TimeGPT-1cited by 158
  22. 2023Towards Better Dynamic Graph Learning: New Architecture and Unified Librarycited by 157
  23. 2022ETSformer: Exponential Smoothing Transformers for Time-series Forecastingcited by 148
  24. 2022FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecastingcited by 147
  25. 2022N-HiTS: Neural Hierarchical Interpolation for Time Series Forecastingcited by 146
  26. 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 144
  27. 2023TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecastingcited by 139
  28. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  29. 2023LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasterscited by 129
  30. 2023FourierGNN: Rethinking Multivariate Time Series Forecasting from a Pure Graph Perspectivecited by 127
  31. 2024Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Seriescited by 127
  32. 2023Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysiscited by 117
  33. 2024TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysiscited by 116
  34. 2023SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modelingcited by 115
  35. 2024Is Mamba Effective for Time Series Forecasting?cited by 114
  36. 2023Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictorscited by 113
  37. 2023Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecastingcited by 113
  38. 2024CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuningcited by 108

50 most cited of 1,676 All 1,676 (JSON)