# Filter then Attend: Improving attention-based Time Series Forecasting with Spectral Filtering

arXiv: https://arxiv.org/abs/2508.20206 · HTML: https://mlsanity.com/paper/2508.20206/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 27 Aug 2025.

## Builds on (most cited first, 19 of 19)

- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [FITS: Modeling Time Series with 10k Parameters](https://mlsanity.com/paper/2307.03756/) (2023)
- [Frequency-domain MLPs are More Effective Learners in Time Series Forecasting](https://mlsanity.com/paper/2311.06184/) (2023)
- [TSLANet: Rethinking Transformers for Time Series Representation Learning](https://mlsanity.com/paper/2404.08472/) (2024)
- [SpectFormer: Frequency and Attention is what you need in a Vision Transformer](https://mlsanity.com/paper/2304.06446/) (2023)
- [FilterNet: Harnessing Frequency Filters for Time Series Forecasting](https://mlsanity.com/paper/2411.01623/) (2024)
- [Fredformer: Frequency Debiased Transformer for Time Series Forecasting](https://mlsanity.com/paper/2406.09009/) (2024)
- [Revitalizing Multivariate Time Series Forecasting: Learnable Decomposition with Inter-Series Dependencies and Intra-Series Variations Modeling](https://mlsanity.com/paper/2402.12694/) (2024)
- [A Survey on Deep Learning based Time Series Analysis with Frequency Transformation](https://mlsanity.com/paper/2302.02173/) (2023)
- [Unlocking the Power of LSTM for Long Term Time Series Forecasting](https://mlsanity.com/paper/2408.10006/) (2024)
- [An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention](https://mlsanity.com/paper/2312.10325/) (2023)
- [Fourier-Mixed Window Attention: Accelerating Informer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2307.00493/) (2023)

## Abstract (arXiv)

Transformer-based models are at the forefront in long time-series forecasting (LTSF). While in many cases, these models are able to achieve state of the art results, they suffer from a bias toward low-frequencies in the data and high computational and memory requirements. Recent work has established that learnable frequency filters can be an integral part of a deep forecasting model by enhancing the model's spectral utilization. These works choose to use a multilayer perceptron to process their filtered signals and thus do not solve the issues found with transformer-based models. In this paper, we establish that adding a filter to the beginning of transformer-based models enhances their performance in long time-series forecasting. We add learnable filters, which only add an additional ≈ 1000 parameters to several transformer-based models and observe in multiple instances 5-10 % relative improvement in forecasting performance. Additionally, we find that with filters added, we are able to decrease the embedding dimension of our models, resulting in transformer-based architectures that are both smaller and more effective than their non-filtering base models. We also conduct synthetic experiments to analyze how the filters enable Transformer-based models to better utilize the full spectrum for forecasting.