# First De-Trend then Attend: Rethinking Attention for Time-Series Forecasting

arXiv: https://arxiv.org/abs/2212.08151 · HTML: https://mlsanity.com/paper/2212.08151/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 24 references that are in the map.
- First posted on arXiv 15 Dec 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [N-BEATS: Neural basis expansion analysis for interpretable time series forecasting](https://mlsanity.com/paper/1905.10437/) (2019)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers](https://mlsanity.com/paper/2111.13587/) (2021)
- [Multiwavelet-based Operator Learning for Differential Equations](https://mlsanity.com/paper/2109.13459/) (2021)
- [ETSformer: Exponential Smoothing Transformers for Time-series Forecasting](https://mlsanity.com/paper/2202.01381/) (2022)
- [FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting](https://mlsanity.com/paper/2205.08897/) (2022)
- [N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting](https://mlsanity.com/paper/2201.12886/) (2022)
- [FreDo: Frequency Domain-based Long-Term Time Series Forecasting](https://mlsanity.com/paper/2205.12301/) (2022)

## Built on by (most cited first, 19 of 19)

- [DSformer: A Double Sampling Transformer for Multivariate Time Series Long-term Prediction](https://mlsanity.com/paper/2308.03274/) (2023)
- [Enhancing Foundation Models for Time Series Forecasting via Wavelet-based Tokenization](https://mlsanity.com/paper/2412.05244/) (2024)
- [Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping](https://mlsanity.com/paper/2305.10721/) (2023)
- [Enhancing Representation Learning for Periodic Time Series with Floss: A Frequency Domain Regularization Approach](https://mlsanity.com/paper/2308.01011/) (2023)
- [Survey and Taxonomy: The Role of Data-Centric AI in Transformer-Based Time Series Forecasting](https://mlsanity.com/paper/2407.19784/) (2024)
- [LiNo: Advancing Recursive Residual Decomposition of Linear and Nonlinear Patterns for Robust Time Series Forecasting](https://mlsanity.com/paper/2410.17159/) (2024)
- [WaveRoRA: Wavelet Rotary Route Attention for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2410.22649/) (2024)
- [D2Vformer: A Flexible Time Series Prediction Model Based on Time Position Embedding](https://mlsanity.com/paper/2409.11024/) (2024)
- [How (Not) to Hybridize Neural and Mechanistic Models for Epidemiological Forecasting](https://mlsanity.com/paper/2602.06323/) (2026)
- [Moving Beyond Functional Connectivity: Time-Series Modeling for fMRI-Based Brain Disorder Classification](https://mlsanity.com/paper/2602.08262/) (2026)
- [Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers](https://mlsanity.com/paper/2305.18382/) (2023)
- [Multi-Scale and Multi-Modal Contrastive Learning Network for Biomedical Time Series](https://mlsanity.com/paper/2312.03796/) (2023)
- [TwinS: Revisiting Non-Stationarity in Multivariate Time Series Forecasting](https://mlsanity.com/paper/2406.03710/) (2024)
- [Revisiting the Seasonal Trend Decomposition for Enhanced Time Series Forecasting](https://mlsanity.com/paper/2602.18465/) (2026)
- [U-shaped Transformer: Retain High Frequency Context in Time Series Analysis](https://mlsanity.com/paper/2307.09019/) (2023)
- [A Novel Spatiotemporal Correlation Anomaly Detection Method Based on Time-Frequency-Domain Feature Fusion and a Dynamic Graph Neural Network in Wireless Sensor Network](https://mlsanity.com/paper/2503.00036/) (2025)
- [Effective Series Decomposition and Components Learning for Time Series Generation](https://mlsanity.com/paper/2511.00747/) (2025)
- [Naga: Vedic Encoding for Deep State Space Models](https://mlsanity.com/paper/2511.13510/) (2025)
- [Revitalizing Medical Time Series with Vision-Informed Retrieval: A Vision-Language Perspective](https://mlsanity.com/paper/2609.34652/) (2026)

## Abstract (arXiv)

Transformer-based models have gained large popularity and demonstrated promising results in long-term time-series forecasting in recent years. In addition to learning attention in time domain, recent works also explore learning attention in frequency domains (e.g., Fourier domain, wavelet domain), given that seasonal patterns can be better captured in these domains. In this work, we seek to understand the relationships between attention models in different time and frequency domains. Theoretically, we show that attention models in different domains are equivalent under linear conditions (i.e., linear kernel to attention scores). Empirically, we analyze how attention models of different domains show different behaviors through various synthetic experiments with seasonality, trend and noise, with emphasis on the role of softmax operation therein. Both these theoretical and empirical analyses motivate us to propose a new method: TDformer (Trend Decomposition Transformer), that first applies seasonal-trend decomposition, and then additively combines an MLP which predicts the trend component with Fourier attention which predicts the seasonal component to obtain the final prediction. Extensive experiments on benchmark time-series forecasting datasets demonstrate that TDformer achieves state-of-the-art performance against existing attention-based models.