mlsanityData to 7 Oct 2026

A Systematic Review for Transformer-based Long-term Series Forecasting

arXiv 2310.20218 ↗First posted 31 Oct 2023cs.LG cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

The emergence of deep learning has yielded noteworthy advancements in time series forecasting (TSF). Transformer architectures, in particular, have witnessed broad utilization and adoption in TSF tasks. Transformers have proven to be the most successful solution to extract the semantic correlations among the elements within a long sequence. Various variants have enabled transformer architecture to effectively handle long-term time series forecasting (LTSF) tasks. In this article, we first present a comprehensive overview of transformer architectures and their subsequent enhancements developed to address various LTSF tasks. Then, we summarize the publicly available LTSF datasets and relevant evaluation metrics. Furthermore, we provide valuable insights into the best practices and techniques for effectively training transformers in the context of time-series analysis. Lastly, we propose potential research directions in this rapidly evolving field.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 57 of its 151 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

57 of 151 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  7. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  8. 2020Reformer: The Efficient Transformercited by 1,658
  9. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  10. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  11. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  12. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,004
38 more
  1. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 869
  2. 2019Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastingcited by 748
  3. 2020A Generalization of Transformer Networks to Graphscited by 653
  4. 2021Transformer in Transformercited by 632
  5. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
  6. 2020Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networkscited by 563
  7. 2020Time Series Forecasting With Deep Learning: A Surveycited by 457
  8. 2020A Transformer-based Framework for Multivariate Time Series Representation Learningcited by 449
  9. 2020A Unifying Review of Deep and Shallow Anomaly Detectioncited by 423
  10. 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 415
  11. 2021Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancycited by 387
  12. 2022Non-stationary Transformers: Exploring the Stationarity in Time Series Forecastingcited by 385
  13. 2016Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Predictioncited by 330
  14. 2017Dilated Residual Networkscited by 293
  15. 2020Transformer Hawkes Processcited by 230
  16. 2015A Critical Review of Recurrent Neural Networks for Sequence Learningcited by 223
  17. 2020Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Predictioncited by 218
  18. 2022TranAD: Deep Transformer Networks for Anomaly Detection in Multivariate Time Series Datacited by 211
  19. 2020Spatial-Temporal Transformer Networks for Traffic Flow Forecastingcited by 179
  20. 2020Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Casecited by 129
  21. 2021Neural Temporal Point Processes: A Reviewcited by 113
  22. 2017Attend and Diagnose: Clinical Time Series Analysis using Attention Modelscited by 105
  23. 2021Learning Graph Structures with Transformer for Multivariate Time Series Anomaly Detection in IoTcited by 104
  24. 2021Transformer Embeddings of Irregularly Spaced Events and Their Participantscited by 90
  25. 2017Dilated Recurrent Neural Networkscited by 83
  26. 2022Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting--Full Versioncited by 59
  27. 2021Gated Transformer Networks for Multivariate Time Series Classificationcited by 58
  28. 2018RobustSTL: A Robust Seasonal-Trend Decomposition Algorithm for Long Time Seriescited by 54
  29. 2017Autoregressive Convolutional Neural Networks for Asynchronous Time Seriescited by 31
  30. 2021Dynamic Gaussian Mixture based Deep Generative Model For Robust Forecasting on Sparse Multivariate Time Seriescited by 30
  31. 2021Learning Interpretable Deep State Space Model for Probabilistic Time Series Forecastingcited by 29
  32. 2021Historical Inertia: A Neglected but Powerful Baseline for Long Sequence Time-series Forecastingcited by 26
  33. 2018A Memory-Network Based Solution for Multivariate Time-Series Forecastingcited by 21
  34. 2020The ASHRAE Great Energy Predictor III competition: Overview and resultscited by 15
  35. 2020Deep Switching Auto-Regressive Factorization:Application to Time Series Forecastingcited by 15
  36. 2021SSDNet: State Space Decomposition Neural Network for Time Series Forecastingcited by 15
  37. 2023Towards Long-Term Time-Series Forecasting: Feature, Pattern, and Distributioncited by 15
  38. 2016A Gentle Tutorial of Recurrent Neural Network with Error Backpropagationcited by 12

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025TFKAN: Time-Frequency KAN for Long-Term Time Series Forecastingcited by 3
  2. 2025Decoding Latent Spaces: Assessing the Interpretability of Time Series Foundation Models for Visual Analyticscited by 2
  3. 2025Explainable time-series forecasting with sampling-free SHAP for Transformerscited by 2
  4. 2025Scaling Transformers for Time Series Forecasting: Do Pretrained Large Models Outperform Small-Scale Alternatives?cited by 0
  5. 2025HN-MVTS: HyperNetwork-based Multivariate Time Series Forecastingcited by 0
  6. 2025Observation-driven correction of numerical weather prediction for marine windscited by 0
  7. 2026DMamba: Decomposition-enhanced Mamba for Time Series Forecastingcited by 0
  8. 2026InfoMamba: An Attention-Free Hybrid Mamba-Transformer Modelcited by 0
  9. 2026IPatch: A Multi-Resolution Transformer Architecture for Robust Time-Series Forecastingcited by 0
  10. 2026Deep Generalised Mixed Models: a Novel Neural Network Structure for Analysing Hierarchical Datacited by 0