mlsanityData to 7 Oct 2026

Local Attention Mechanism: Boosting the Transformer Architecture for Long-Sequence Time Series Forecasting

arXiv 2410.03805 ↗First posted 4 Oct 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Transformers have become the leading choice in natural language processing over other deep learning architectures. This trend has also permeated the field of time series analysis, especially for long-horizon forecasting, showcasing promising results both in performance and running time. In this paper, we introduce Local Attention Mechanism (LAM), an efficient attention mechanism tailored for time series analysis. This mechanism exploits the continuity properties of time series to reduce the number of attention scores computed. We present an algorithm for implementing LAM in tensor algebra that runs in time and memory O(nlogn), significantly improving upon the O(n^2) time and memory complexity of traditional attention mechanisms. We also note the lack of proper datasets to evaluate long-horizon forecast models. Thus, we propose a novel set of datasets to improve the evaluation of models addressing long-horizon forecasting challenges. Our experimental analysis demonstrates that the vanilla transformer architecture magnified with LAM surpasses state-of-the-art models, including the vanilla attention mechanism. These results confirm the effectiveness of our approach and highlight a range of future challenges in long-sequence time series forecasting.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 53 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

20 of 53 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2020Reformer: The Efficient Transformercited by 1,658
  5. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  6. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  7. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  8. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,004
  9. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 869
  10. 2019Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecastingcited by 748
  11. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
  12. 2019Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019cited by 270
8 more
  1. 2015A Critical Review of Recurrent Neural Networks for Sequence Learningcited by 223
  2. 2019Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directionscited by 172
  3. 2019Attention in Natural Language Processingcited by 100
  4. 2020Limitations of Autoregressive Models and Their Alternativescited by 54
  5. 2020A Mathematical Theory of Attentioncited by 43
  6. 2023A Comprehensive Survey of Deep Transfer Learning for Anomaly Detection in Industrial Time Series: Methods, Applications, and Directionscited by 22
  7. 2023Deep learning models for price forecasting of financial time series: A review of recent advancements: 2020-2022cited by 16
  8. 2024Attention as an RNNcited by 6

Built on by

Not cited yet.