mlsanityData to 14 Oct 2026

Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting

arXiv 2205.14415 ↗First posted 28 May 2022cs.LG eess.SPcited by 391

Not on the map. Why?

Abstract (arXiv)

Transformers have shown great power in time series forecasting due to their global-range modeling ability. However, their performance can degenerate terribly on non-stationary real-world data in which the joint distribution changes over time. Previous studies primarily adopt stationarization to attenuate the non-stationarity of original series for better predictability. But the stationarized series deprived of inherent non-stationarity can be less instructive for real-world bursty events forecasting. This problem, termed over-stationarization in this paper, leads Transformers to generate indistinguishable temporal attentions for different series and impedes the predictive capability of deep models. To tackle the dilemma between series predictability and model capability, we propose Non-stationary Transformers as a generic framework with two interdependent modules: Series Stationarization and De-stationary Attention. Concretely, Series Stationarization unifies the statistics of each input and converts the output with restored statistics for better predictability. To address the over-stationarization problem, De-stationary Attention is devised to recover the intrinsic non-stationary information into temporal dependencies by approximating distinguishable attentions learned from raw series. Our Non-stationary Transformers framework consistently boosts mainstream Transformers by a large margin, which reduces MSE by 49.43% on Transformer, 47.34% on Informer, and 46.89% on Reformer, making them the state-of-the-art in time series forecasting. Code is available at this repository: https://github.com/thuml/Nonstationary_Transformers.

Key facts. Cited by 391 papers in the map (data through 14 Oct 2026). Builds on 25 of its 48 references that are in the map. First posted on arXiv 28 May 2022.

Builds on

25 of 48 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  8. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  9. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
  10. 2020Reformer: The Efficient Transformercited by 1,668
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  12. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
13 more
  1. 2016Instance Normalization: The Missing Ingredient for Fast Stylizationcited by 1,198
  2. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,013
  3. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  4. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 879
  5. 2019N-BEATS: Neural basis expansion analysis for interpretable time series forecastingcited by 753
  6. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  7. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 607
  8. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  9. 2017A Multi-Horizon Quantile Recurrent Forecastercited by 211
  10. 2022ETSformer: Exponential Smoothing Transformers for Time-series Forecastingcited by 148
  11. 2022N-HiTS: Neural Hierarchical Interpolation for Time Series Forecastingcited by 146
  12. 2017Long-term Forecasting using Higher Order Tensor RNNscited by 34
  13. 2018Deep Factors with Gaussian Processes for Forecastingcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 31; 2024: 113; 2025: 124; 2026: 118.

391 papers.

Sort:

By area:

  1. 2023Time-LLM: Time Series Forecasting by Reprogramming Large Language Modelscited by 700
  2. 2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 632
  3. 2024TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methodscited by 249
  4. 2024TimeXer: Empowering Transformers for Time Series Forecasting with Exogenous Variablescited by 248
  5. 2023TSMixer: An All-MLP Architecture for Time Series Forecastingcited by 240
  6. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 197
  7. 2025Sundial: A Family of Highly Capable Time Series Foundation Modelscited by 180
  8. 2023TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecastingcited by 174
  9. 2023TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Seriescited by 169
  10. 2023UniTime: A Language-Empowered Unified Model for Cross-Domain Time Series Forecastingcited by 144
  11. 2024TimeMixer++: A General Time Series Pattern Machine for Universal Predictive Analysiscited by 116
  12. 2023SimMTM: A Simple Pre-Training Framework for Masked Time-Series Modelingcited by 115
38 more
  1. 2023Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecastingcited by 113
  2. 2024CycleNet: Enhancing Time Series Forecasting through Modeling Periodic Patternscited by 106
  3. 2024Moirai-MoE: Empowering Time Series Foundation Models with Sparse Mixture of Expertscited by 95
  4. 2024VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time Series Forecasterscited by 88
  5. 2024Timer-XL: Long-Context Transformers for Unified Time Series Forecastingcited by 87
  6. 2024SOFTS: Efficient Multivariate Time Series Forecasting with Series-Core Fusioncited by 76
  7. 2024Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysiscited by 74
  8. 2024AutoTimes: Autoregressive Time Series Forecasters via Large Language Modelscited by 65
  9. 2024TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecastingcited by 64
  10. 2025Time-VLM: Exploring Multimodal Vision-Language Models for Augmented Time Series Forecastingcited by 63
  11. 2023CARD: Channel Aligned Robust Blend Transformer for Time Series Forecastingcited by 55
  12. 2024Medformer: A Multi-Granularity Patching Transformer for Medical Time-Series Classificationcited by 53
  13. 2024Frequency Adaptive Normalization For Non-stationary Time Series Forecastingcited by 47
  14. 2024FilterNet: Harnessing Frequency Filters for Time Series Forecastingcited by 47
  15. 2024Revitalizing Multivariate Time Series Forecasting: Learnable Decomposition with Inter-Series Dependencies and Intra-Series Variations Modelingcited by 44
  16. 2024TimeBridge: Non-Stationarity Matters for Long-term Time Series Forecastingcited by 40
  17. 2024TOTEM: TOkenized Time Series EMbeddings for General Time Series Analysiscited by 39
  18. 2025Context-Alignment: Activating and Enhancing LLM Capabilities in Time Seriescited by 39
  19. 2025Enhancing Time Series Forecasting through Selective Representation Spaces: A Patch Perspectivecited by 39
  20. 2024SensorLLM: Aligning Large Language Models with Motion Sensors for Human Activity Recognitioncited by 38
  21. 2024TSI-Bench: Benchmarking Time Series Imputationcited by 33
  22. 2025Temporal Query Network for Efficient Multivariate Time Series Forecastingcited by 31
  23. 2024Zero-shot forecasting of chaotic systemscited by 28
  24. 2026Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scalingcited by 28
  25. 2024Time-Series Forecasting for Out-of-Distribution Generalization Using Invariant Learningcited by 27
  26. 2025Non-stationary Diffusion For Probabilistic Time Series Forecastingcited by 27
  27. 2024Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learningcited by 25
  28. 2024Learning Pattern-Specific Experts for Time Series Forecasting Under Patch-level Distribution Shiftcited by 24
  29. 2023Pushing the Limits of Pre-training for Time Series Forecasting in the CloudOps Domaincited by 23
  30. 2024RWKV-TS: Beyond Traditional Recurrent Neural Network for Time Series Taskscited by 22
  31. 2024Are Self-Attentions Effective for Time Series Forecasting?cited by 19
  32. 2024xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memoriescited by 19
  33. 2024Proactive Model Adaptation Against Concept Drift for Online Time Series Forecastingcited by 19
  34. 2022Learning Deep Time-index Models for Time Series Forecastingcited by 18
  35. 2024BACKTIME: Backdoor Attacks on Multivariate Time Series Forecastingcited by 18
  36. 2025FreEformer: Frequency Enhanced Transformer for Multivariate Time Series Forecastingcited by 18
  37. 2024Attractor Memory for Long-Term Time Series Forecasting: A Chaos Perspectivecited by 17
  38. 2024S2IP-LLM: Semantic Space Informed Prompt Learning with LLM for Time Series Forecastingcited by 17

50 most cited of 391 All 391 (JSON)