# Deep Autoregressive Models with Spectral Attention

arXiv: https://arxiv.org/abs/2107.05984 · HTML: https://mlsanity.com/paper/2107.05984/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 73 references that are in the map.
- First posted on arXiv 13 Jul 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Generating Wikipedia by Summarizing Long Sequences](https://mlsanity.com/paper/1801.10198/) (2018)
- [Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting](https://mlsanity.com/paper/2103.07719/) (2021)
- [Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019](https://mlsanity.com/paper/1911.13288/) (2019)
- [Temporal Convolutional Networks: A Unified Approach to Action Segmentation](https://mlsanity.com/paper/1608.08242/) (2016)
- [Transformer Networks for Trajectory Forecasting](https://mlsanity.com/paper/2003.08111/) (2020)
- [Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case](https://mlsanity.com/paper/2001.08317/) (2020)
- [Deep Factors for Forecasting](https://mlsanity.com/paper/1905.12417/) (2019)
- [GluonTS: Probabilistic Time Series Models in Python](https://mlsanity.com/paper/1906.05264/) (2019)
- [GRATIS: GeneRAting TIme Series with diverse and controllable characteristics](https://mlsanity.com/paper/1903.02787/) (2019)
- [Language Through a Prism: A Spectral Approach for Multiscale Language Representations](https://mlsanity.com/paper/2011.04823/) (2020)
- [A Hybrid Residual Dilated LSTM end Exponential Smoothing Model for Mid-Term Electric Load Forecasting](https://mlsanity.com/paper/2004.00508/) (2020)
- [Spatiotemporal Adaptive Neural Network for Long-term Forecasting of Financial Time Series](https://mlsanity.com/paper/2003.12194/) (2020)

## Built on by (most cited first, 9 of 9)

- [Irregular Traffic Time Series Forecasting Based on Asynchronous Spatio-Temporal Graph Convolutional Network](https://mlsanity.com/paper/2308.16818/) (2023)
- [Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patching](https://mlsanity.com/paper/2405.20799/) (2024)
- [Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers](https://mlsanity.com/paper/2302.01925/) (2023)
- [From Pixels to Predictions: Spectrogram and Vision Transformer for Better Time Series Forecasting](https://mlsanity.com/paper/2403.11047/) (2024)
- [DeepVol: Volatility Forecasting from High-Frequency Data with Dilated Causal Convolutions](https://mlsanity.com/paper/2210.04797/) (2022)
- [Rough Transformers for Continuous and Efficient Time-Series Modelling](https://mlsanity.com/paper/2403.10288/) (2024)
- [Training Implicit Generative Models via an Invariant Statistical Loss](https://mlsanity.com/paper/2402.16435/) (2024)
- [Transformer Vibration Forecasting for Advancing Rail Safety and Maintenance 4.0](https://mlsanity.com/paper/2501.11730/) (2025)
- [Interpretable Spectral Variational AutoEncoder (ISVAE) for time series clustering](https://mlsanity.com/paper/2310.11940/) (2023)

## Abstract (arXiv)

Time series forecasting is an important problem across many domains, playing a crucial role in multiple real-world applications. In this paper, we propose a forecasting architecture that combines deep autoregressive models with a Spectral Attention (SA) module, which merges global and local frequency domain information in the model's embedded space. By characterizing in the spectral domain the embedding of the time series as occurrences of a random process, our method can identify global trends and seasonality patterns. Two spectral attention models, global and local to the time series, integrate this information within the forecast and perform spectral filtering to remove time series's noise. The proposed architecture has a number of useful properties: it can be effectively incorporated into well-know forecast architectures, requiring a low number of parameters and producing interpretable results that improve forecasting accuracy. We test the Spectral Attention Autoregressive Model (SAAM) on several well-know forecast datasets, consistently demonstrating that our model compares favorably to state-of-the-art approaches.