mlsanityData to 7 Oct 2026

Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting

arXiv 1907.00235 ↗First posted 29 Jun 2019cs.LG stat.MLcited by 606

Not on the map. Why?

Abstract (arXiv)

Time series forecasting is an important problem across many domains, including predictions of solar plant energy output, electricity consumption, and traffic jam situation. In this paper, we propose to tackle such forecasting problem with Transformer [1]. Although impressed by its performance in our preliminary study, we found its two major weaknesses: (1) locality-agnostics: the point-wise dot-product self-attention in canonical Transformer architecture is insensitive to local context, which can make the model prone to anomalies in time series; (2) memory bottleneck: space complexity of canonical Transformer grows quadratically with sequence length L, making directly modeling long time series infeasible. In order to solve these two issues, we first propose convolutional self-attention by producing queries and keys with causal convolution so that local context can be better incorporated into attention mechanism. Then, we propose LogSparse Transformer with only O(L(log L)2) memory cost, improving forecasting accuracy for time series with fine granularity and strong long-term dependencies under constrained memory budget. Our experiments on both synthetic data and real-world datasets show that it compares favorably to the state-of-the-art.

Key facts. Cited by 606 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 80; 2022: 111; 2023: 145; 2024: 131; 2025: 91; 2026: 48.

606 papers.

Sort:

By area:

  1. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
  2. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  3. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  4. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  5. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  6. 2022TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysiscited by 1,030
  7. 2022Transformers in Time Series: A Surveycited by 489
  8. 2021TS2Vec: Towards Universal Representation of Time Seriescited by 463
  9. 2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
  10. 2021Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancycited by 387
  11. 2022Non-stationary Transformers: Exploring the Stationarity in Time Series Forecastingcited by 385
  12. 2021Random Feature Attentioncited by 350
38 more
  1. 2021SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interactioncited by 335
  2. 2021Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecastingcited by 332
  3. 2023Long-term Forecasting with TiDE: Time-series Dense Encodercited by 327
  4. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 308
  5. 2021Monash Time Series Forecasting Archivecited by 256
  6. 2024TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variablescited by 241
  7. 2023TSMixer: An All-MLP Architecture for Time Series Forecastingcited by 232
  8. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  9. 2022Diffusion-based Time Series Imputation and Forecasting with Structured State Space Modelscited by 195
  10. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 191
  11. 2022Transformer Quality in Linear Timecited by 189
  12. 2022PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecastingcited by 170
  13. 2021Discrete Graph Structure Learning for Forecasting Multiple Time Seriescited by 169
  14. 2022Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structurescited by 158
  15. 2023Towards Better Dynamic Graph Learning: New Architecture and Unified Librarycited by 154
  16. 2022FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecastingcited by 147
  17. 2022N-HiTS: Neural Hierarchical Interpolation for Time Series Forecastingcited by 146
  18. 2022ETSformer: Exponential Smoothing Transformers for Time-series Forecastingcited by 146
  19. 2022CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecastingcited by 143
  20. 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 139
  21. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  22. 2023Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysiscited by 113
  23. 2024TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysiscited by 113
  24. 2024Is Mamba Effective for Time Series Forecasting?cited by 110
  25. 2023DCdetector: Dual Attention Contrastive Representation Learning for Time Series Anomaly Detectioncited by 103
  26. 2023Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecastingcited by 102
  27. 2024Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecastingcited by 102
  28. 2024CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patternscited by 102
  29. 2024UniTS: A Unified Multi-Task Time Series Modelcited by 93
  30. 2024From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflectioncited by 93
  31. 2023Generative Time Series Forecasting with Diffusion, Denoise, and Disentanglementcited by 89
  32. 2022Spatio-Temporal Meta-Graph Learning for Traffic Forecastingcited by 88
  33. 2021Sparse is Enough in Scaling Transformerscited by 86
  34. 2024Timer-XL: Long-Context Transformers for Unified Time Series Forecastingcited by 83
  35. 2023Time Series as Images: Vision Transformer for Irregularly Sampled Time Seriescited by 79
  36. 2023Non-autoregressive Conditional Diffusion Models for Time Series Predictioncited by 77
  37. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  38. 2021DL-Traff: Survey and Benchmark of Deep Learning Models for Urban Traffic Predictioncited by 73

50 most cited of 606 All 606 (JSON)