mlsanityData to 7 Oct 2026

A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

arXiv 2211.14730 ↗First posted 27 Nov 2022cs.LG cs.AIcited by 1,981

Not on the map. Why?

Cited by 48 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose an efficient design of Transformer-based models for multivariate time series forecasting and self-supervised representation learning. It is based on two key components: (i) segmentation of time series into subseries-level patches which are served as input tokens to Transformer; (ii) channel-independence where each channel contains a single univariate time series that shares the same embedding and Transformer weights across all the series. Patching design naturally has three-fold benefit: local semantic information is retained in the embedding; computation and memory usage of the attention maps are quadratically reduced given the same look-back window; and the model can attend longer history. Our channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of SOTA Transformer-based models. We also apply our model to self-supervised pre-training tasks and attain excellent fine-tuning performance, which outperforms supervised training on large datasets. Transferring of masked pre-trained representation on one dataset to others also produces SOTA forecasting accuracy. Code is available at: https://github.com/yuqinie98/PatchTST.

Key facts. Cited by 1,981 papers in the map (data through 7 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 27 Nov 2022.

Builds on

33 of 45 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  10. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  11. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  12. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
21 more
  1. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  2. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  3. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,187
  4. 2021Transformers in Vision: A Surveycited by 1,009
  5. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,004
  6. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  7. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
  8. 2022Transformers in Time Series: A Surveycited by 489
  9. 2021TS2Vec: Towards Universal Representation of Time Seriescited by 463
  10. 2020Time Series Forecasting With Deep Learning: A Surveycited by 457
  11. 2020A Transformer-based Framework for Multivariate Time Series Representation Learningcited by 449
  12. 2021Time-Series Representation Learning via Temporal and Contextual Contrastingcited by 342
  13. 2021Spectral Temporal Graph Neural Network for Multivariate Time-series Forecastingcited by 298
  14. 2019Unsupervised Scalable Representation Learning for Multivariate Time Seriescited by 267
  15. 2021Unsupervised Representation Learning for Time Series with Temporal Neighborhood Codingcited by 223
  16. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  17. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  18. 2022Unsupervised Time-Series Representation Learning with Iterative Bilinear Temporal-Spectral Fusioncited by 96
  19. 2021Z-GCNETs: Time Zigzags at Graph Convolutional Networks for Time Series Forecastingcited by 67
  20. 2022Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting--Full Versioncited by 59
  21. 2021An Experimental Review on Deep Learning Architectures for Time Series Forecastingcited by 47

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 126; 2024: 461; 2025: 660; 2026: 733.

1,981 papers.

Sort:

By area:

  1. 2023iTransformer: Inverted Transformers Are Effective for Time Series Forecastingcited by 906
  2. 2024Chronos: Learning the Language of Time Seriescited by 712
  3. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 680
  4. 2023A decoder-only foundation model for time-series forecastingcited by 650
  5. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
  6. 2024Unified Training of Universal Time Series Forecasting Transformerscited by 588
  7. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
  8. 2024TimeMixer: Decomposable Multiscale Mixing for Time Series Forecastingcited by 333
  9. 2023Long-term Forecasting with TiDE: Time-series Dense Encodercited by 327
  10. 2024Titans: Learning to Memorize at Test Timecited by 303
  11. 2024Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Expertscited by 280
  12. 2024TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methodscited by 242
38 more
  1. 2024TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variablescited by 241
  2. 2023TSMixer: An All-MLP Architecture for Time Series Forecastingcited by 232
  3. 2024Timer: Generative Pre-trained Transformers Are Large Time Series Modelscited by 198
  4. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 191
  5. 2024GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluationcited by 188
  6. 2024CBraMod: A Criss-Cross Brain Foundation Model for EEG Decodingcited by 188
  7. 2023FITS: Modeling Time Series with 10k Parameterscited by 185
  8. 2023TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecastingcited by 170
  9. 2023Frequency-domain MLPs are More Effective Learners in Time Series Forecastingcited by 170
  10. 2025Sundial: A Family of Highly Capable Time Series Foundation Modelscited by 168
  11. 2023TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Seriescited by 165
  12. 2024Are Language Models Actually Useful for Time Series Forecasting?cited by 163
  13. 2025Chronos-2: From Univariate to Universal Forecastingcited by 158
  14. 2023Towards Better Dynamic Graph Learning: New Architecture and Unified Librarycited by 154
  15. 2023TimeGPT-1cited by 149
  16. 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 139
  17. 2023TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecastingcited by 136
  18. 2023LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasterscited by 127
  19. 2024Tiny Time Mixers (TTMs): Fast Pre-trained Models for Enhanced Zero/Few-Shot Forecasting of Multivariate Time Seriescited by 120
  20. 2023Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysiscited by 113
  21. 2024Large Language Models for Time Series: A Surveycited by 113
  22. 2024TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysiscited by 113
  23. 2023SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modelingcited by 112
  24. 2023Koopa: Learning Non-stationary Time Series Dynamics with Koopman Predictorscited by 110
  25. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  26. 2025TiRex: Zero-Shot Forecasting Across Long and Short Horizons with Enhanced In-Context Learningcited by 107
  27. 2024CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuningcited by 105
  28. 2024ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilitiescited by 104
  29. 2024DUET: Dual Clustering Enhanced Multivariate Time Series Forecastingcited by 104
  30. 2023DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detectioncited by 103
  31. 2024SparseTSF: Modeling Long-term Time Series Forecasting with 1k Parameterscited by 103
  32. 2023Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecastingcited by 102
  33. 2024Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecastingcited by 102
  34. 2024Approaching Human-Level Forecasting with Language Modelscited by 102
  35. 2024CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patternscited by 102
  36. 2023The Capacity and Robustness Trade-off: Revisiting the Channel Independent Strategy for Multivariate Time Series Forecastingcited by 100
  37. 2023Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospectscited by 97
  38. 2024UniTS: A Unified Multi-Task Time Series Modelcited by 93

50 most cited of 1,981 All 1,981 (JSON)