# The Power of Architecture: Deep Dive into Transformer Architectures for Long-Term Time Series Forecasting

arXiv: https://arxiv.org/abs/2507.13043 · HTML: https://mlsanity.com/paper/2507.13043/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 17 Jul 2025.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis](https://mlsanity.com/paper/2210.02186/) (2022)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting](https://mlsanity.com/paper/1912.09363/) (2019)
- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods](https://mlsanity.com/paper/2403.20150/) (2024)
- [Timer: Generative Pre-trained Transformers Are Large Time Series Models](https://mlsanity.com/paper/2402.02368/) (2024)
- [What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?](https://mlsanity.com/paper/2204.05832/) (2022)
- [ETSformer: Exponential Smoothing Transformers for Time-series Forecasting](https://mlsanity.com/paper/2202.01381/) (2022)
- [PDFormer: Propagation Delay-Aware Dynamic Long-Range Transformer for Traffic Flow Prediction](https://mlsanity.com/paper/2301.07945/) (2023)
- [Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis](https://mlsanity.com/paper/2310.06119/) (2023)
- [Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting](https://mlsanity.com/paper/2402.05956/) (2024)
- [TSLANet: Rethinking Transformers for Time Series Representation Learning](https://mlsanity.com/paper/2404.08472/) (2024)
- [Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2311.12351/) (2023)
- [Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting](https://mlsanity.com/paper/2206.04038/) (2022)
- [Fredformer: Frequency Debiased Transformer for Time Series Forecasting](https://mlsanity.com/paper/2406.09009/) (2024)
- [Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder](https://mlsanity.com/paper/2304.04052/) (2023)
- [BasisFormer: Attention-based Time Series Forecasting with Learnable and Interpretable Basis](https://mlsanity.com/paper/2310.20496/) (2023)
- [Are Self-Attentions Effective for Time Series Forecasting?](https://mlsanity.com/paper/2405.16877/) (2024)

## Abstract (arXiv)

Transformer-based models have recently become dominant in Long-term Time Series Forecasting (LTSF), yet the variations in their architecture, such as encoder-only, encoder-decoder, and decoder-only designs, raise a crucial question: What Transformer architecture works best for LTSF tasks? However, existing models are often tightly coupled with various time-series-specific designs, making it difficult to isolate the impact of the architecture itself. To address this, we propose a novel taxonomy that disentangles these designs, enabling clearer and more unified comparisons of Transformer architectures. Our taxonomy considers key aspects such as attention mechanisms, forecasting aggregations, forecasting paradigms, and normalization layers. Through extensive experiments, we uncover several key insights: bi-directional attention with joint-attention is most effective; more complete forecasting aggregation improves performance; and the direct-mapping paradigm outperforms autoregressive approaches. Furthermore, our combined model, utilizing optimal architectural choices, consistently outperforms several existing models, reinforcing the validity of our conclusions. We hope these findings offer valuable guidance for future research on Transformer architectural designs in LTSF. Our code is available at https://github.com/HALF111/TSF_architecture.