# Local Attention Mechanism: Boosting the Transformer Architecture for Long-Sequence Time Series Forecasting

arXiv: https://arxiv.org/abs/2410.03805 · HTML: https://mlsanity.com/paper/2410.03805/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 53 references that are in the map.
- First posted on arXiv 4 Oct 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019](https://mlsanity.com/paper/1911.13288/) (2019)
- [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://mlsanity.com/paper/1506.00019/) (2015)
- [Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions](https://mlsanity.com/paper/1909.00590/) (2019)
- [Attention in Natural Language Processing](https://mlsanity.com/paper/1902.02181/) (2019)
- [Limitations of Autoregressive Models and Their Alternatives](https://mlsanity.com/paper/2010.11939/) (2020)
- [A Mathematical Theory of Attention](https://mlsanity.com/paper/2007.02876/) (2020)
- [A Comprehensive Survey of Deep Transfer Learning for Anomaly Detection in Industrial Time Series: Methods, Applications, and Directions](https://mlsanity.com/paper/2307.05638/) (2023)
- [Deep learning models for price forecasting of financial time series: A review of recent advancements: 2020-2022](https://mlsanity.com/paper/2305.04811/) (2023)
- [Attention as an RNN](https://mlsanity.com/paper/2405.13956/) (2024)

## Built on by (most cited first, 1 of 1)

- [Interpolation between Convolution and Attention via K-Nearest Neighbors](https://mlsanity.com/paper/2606.14725/) (2026)

## Abstract (arXiv)

Transformers have become the leading choice in natural language processing over other deep learning architectures. This trend has also permeated the field of time series analysis, especially for long-horizon forecasting, showcasing promising results both in performance and running time. In this paper, we introduce Local Attention Mechanism (LAM), an efficient attention mechanism tailored for time series analysis. This mechanism exploits the continuity properties of time series to reduce the number of attention scores computed. We present an algorithm for implementing LAM in tensor algebra that runs in time and memory O(nlogn), significantly improving upon the O(n^2) time and memory complexity of traditional attention mechanisms. We also note the lack of proper datasets to evaluate long-horizon forecast models. Thus, we propose a novel set of datasets to improve the evaluation of models addressing long-horizon forecasting challenges. Our experimental analysis demonstrates that the vanilla transformer architecture magnified with LAM surpasses state-of-the-art models, including the vanilla attention mechanism. These results confirm the effectiveness of our approach and highlight a range of future challenges in long-sequence time series forecasting.