# Are Self-Attentions Effective for Time Series Forecasting?

arXiv: https://arxiv.org/abs/2405.16877 · HTML: https://mlsanity.com/paper/2405.16877/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 27 May 2024.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [A decoder-only foundation model for time-series forecasting](https://mlsanity.com/paper/2310.10688/) (2023)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [MOMENT: A Family of Open Time-series Foundation Models](https://mlsanity.com/paper/2402.03885/) (2024)
- [Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting](https://mlsanity.com/paper/2205.14415/) (2022)
- [TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting](https://mlsanity.com/paper/2405.14616/) (2024)
- [TSMixer: An All-MLP Architecture for Time Series Forecasting](https://mlsanity.com/paper/2303.06053/) (2023)
- [FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting](https://mlsanity.com/paper/2205.08897/) (2022)
- [TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2306.09364/) (2023)
- [Lite Vision Transformer with Enhanced Self-Attention](https://mlsanity.com/paper/2112.10809/) (2021)
- [An Evaluation of Change Point Detection Algorithms](https://mlsanity.com/paper/2003.06222/) (2020)
- [Learning Deep Time-index Models for Time Series Forecasting](https://mlsanity.com/paper/2207.06046/) (2022)
- [Fcaformer: Forward Cross Attention in Hybrid Vision Transformer](https://mlsanity.com/paper/2211.07198/) (2022)

## Built on by (most cited first, 19 of 19)

- [Rethinking Irregular Time Series Forecasting: A Simple yet Effective Baseline](https://mlsanity.com/paper/2505.11250/) (2025)
- [Temporal Query Network for Efficient Multivariate Time Series Forecasting](https://mlsanity.com/paper/2505.12917/) (2025)
- [Why Attention Fails: The Degeneration of Transformers into MLPs in Time Series Forecasting](https://mlsanity.com/paper/2509.20942/) (2025)
- [DropoutTS: Sample-Adaptive Dropout for Robust Time Series Forecasting](https://mlsanity.com/paper/2601.21726/) (2026)
- [Towards Robust Real-World Multivariate Time Series Forecasting: A Unified Framework for Dependency, Asynchrony, and Missingness](https://mlsanity.com/paper/2506.08660/) (2025)
- [VARMA-Enhanced Transformer for Time Series Forecasting](https://mlsanity.com/paper/2509.04782/) (2025)
- [ARIES: Relation Assessment and Model Recommendation for Deep Time Series Forecasting](https://mlsanity.com/paper/2509.06060/) (2025)
- [TimePerceiver: An Encoder-Decoder Framework for Generalized Time-Series Forecasting](https://mlsanity.com/paper/2512.22550/) (2025)
- [A HEART for the environment: Transformer-Based Spatiotemporal Modeling for Air Quality Prediction](https://mlsanity.com/paper/2502.19042/) (2025)
- [Efficient Time Series Forecasting via Hyper-Complex Models and Frequency Aggregation](https://mlsanity.com/paper/2502.19983/) (2025)
- [The Power of Architecture: Deep Dive into Transformer Architectures for Long-Term Time Series Forecasting](https://mlsanity.com/paper/2507.13043/) (2025)
- [Transforming Credit Risk Analysis: A Time-Series-Driven ResE-BiLSTM Framework for Post-Loan Default Detection](https://mlsanity.com/paper/2508.00415/) (2025)
- [STEPS: A Temporal Smooth Error Propagation Solver on the Manifolds for Test-Time Adaptation in Time Series Forecasting](https://mlsanity.com/paper/2605.08005/) (2026)
- [Physics-Guided Spatiotemporal State Space Modeling for Lookahead Molten Pool Segmentation in Laser Wire-Feed Welding](https://mlsanity.com/paper/2606.23028/) (2026)
- [Self-Gating Attention for Efficient Time Series Forecasting](https://mlsanity.com/paper/2607.02344/) (2026)
- [MnemoDyn: Learning Resting State Dynamics from 40K FMRI sequences](https://mlsanity.com/paper/2608.23936/) (2026)
- [Beyond Average Error through Oracle-Informed Stress Tests for Time-Series Forecasting](https://mlsanity.com/paper/2609.22820/) (2026)
- [Pooling Helps, Learned Weighting Hurts In-Context: Decomposing Group Attention](https://mlsanity.com/paper/2610.01831/) (2026)
- [Beyond Static Positional Encoding: Bayesian Time-aware Attention for Non-Stationary Long-term Time Series Forecasting](https://mlsanity.com/paper/2610.19603/) (2026)

## Abstract (arXiv)

Time series forecasting is crucial for applications across multiple domains and various scenarios. Although Transformer models have dramatically advanced the landscape of forecasting, their effectiveness remains debated. Recent findings have indicated that simpler linear models might outperform complex Transformer-based approaches, highlighting the potential for more streamlined architectures. In this paper, we shift the focus from evaluating the overall Transformer architecture to specifically examining the effectiveness of self-attention for time series forecasting. To this end, we introduce a new architecture, Cross-Attention-only Time Series transformer (CATS), that rethinks the traditional Transformer framework by eliminating self-attention and leveraging cross-attention mechanisms instead. By establishing future horizon-dependent parameters as queries and enhanced parameter sharing, our model not only improves long-term forecasting accuracy but also reduces the number of parameters and memory usage. Extensive experiment across various datasets demonstrates that our model achieves superior performance with the lowest mean squared error and uses fewer parameters compared to existing models. The implementation of our model is available at: https://github.com/dongbeank/CATS.