# Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting

arXiv: https://arxiv.org/abs/2410.20772 · HTML: https://mlsanity.com/paper/2410.20772/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 55 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction](https://mlsanity.com/paper/2106.09305/) (2021)
- [Long-term Forecasting with TiDE: Time-series Dense Encoder](https://mlsanity.com/paper/2304.08424/) (2023)
- [FITS: Modeling Time Series with 10k Parameters](https://mlsanity.com/paper/2307.03756/) (2023)
- [Frequency-domain MLPs are More Effective Learners in Time Series Forecasting](https://mlsanity.com/paper/2311.06184/) (2023)
- [Deep Learning using Rectified Linear Units (ReLU)](https://mlsanity.com/paper/1803.08375/) (2018)
- [Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping](https://mlsanity.com/paper/2305.10721/) (2023)

## Built on by (most cited first, 3 of 3)

- [WaveRoRA: Wavelet Rotary Route Attention for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2410.22649/) (2024)
- [Theoretical Foundation of Flow-Based Time Series Generation: Provable Approximation, Generalization, and Efficiency](https://mlsanity.com/paper/2503.14076/) (2025)
- [BeatFormer: Efficient motion-robust remote heart rate estimation through unsupervised spectral zoomed attention filters](https://mlsanity.com/paper/2507.14885/) (2025)

## Abstract (arXiv)

Sequence modeling faces challenges in capturing long-range dependencies across diverse tasks. Recent linear and transformer-based forecasters have shown superior performance in time series forecasting. However, they are constrained by their inherent inability to effectively address long-range dependencies in time series data, primarily due to using fixed-size inputs for prediction. Furthermore, they typically sacrifice essential temporal correlation among consecutive training samples by shuffling them into mini-batches. To overcome these limitations, we introduce a fast and effective Spectral Attention mechanism, which preserves temporal correlations among samples and facilitates the handling of long-range information while maintaining the base model structure. Spectral Attention preserves long-period trends through a low-pass filter and facilitates gradient to flow between samples. Spectral Attention can be seamlessly integrated into most sequence models, allowing models with fixed-sized look-back windows to capture long-range dependencies over thousands of steps. Through extensive experiments on 11 real-world time series datasets using 7 recent forecasting models, we consistently demonstrate the efficacy of our Spectral Attention mechanism, achieving state-of-the-art results.