mlsanityData to 7 Oct 2026

Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting

arXiv 2206.04038 ↗First posted 8 Jun 2022cs.LGcited by 54

Not on the map. Why?

Abstract (arXiv)

The performance of time series forecasting has recently been greatly improved by the introduction of transformers. In this paper, we propose a general multi-scale framework that can be applied to the state-of-the-art transformer-based time series forecasting models (FEDformer, Autoformer, etc.). By iteratively refining a forecasted time series at multiple scales with shared weights, introducing architecture adaptations, and a specially-designed normalization scheme, we are able to achieve significant performance improvements, from 5.5% to 38.5% across datasets and transformer architectures, with minimal additional computational overhead. Via detailed ablation studies, we demonstrate the effectiveness of each of our contributions across the architecture and methodology. Furthermore, our experiments on various public datasets demonstrate that the proposed improvements outperform their corresponding baseline counterparts. Our code is publicly available in https://github.com/BorealisAI/scaleformer.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 27 of its 50 references that are in the map. First posted on arXiv 8 Jun 2022.

Builds on

27 of 50 references.

References · 27

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
  4. 2020Reformer: The Efficient Transformercited by 1,658
  5. 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,654
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
  8. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  9. 2017DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networkscited by 1,004
  10. 2021Multiscale Vision Transformerscited by 896
  11. 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 869
  12. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 606
15 more
  1. 2020A Transformer-based Framework for Multivariate Time Series Representation Learningcited by 449
  2. 2017A General and Adaptive Robust Loss Functioncited by 236
  3. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  4. 2018Multi-level Wavelet-CNN for Image Restorationcited by 184
  5. 2022FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecastingcited by 147
  6. 2022N-HiTS: Neural Hierarchical Interpolation for Time Series Forecastingcited by 146
  7. 2020Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Casecited by 129
  8. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
  9. 2016Multi-Scale Convolutional Neural Networks for Time Series Classificationcited by 99
  10. 2021Hierarchical Transformers Are More Efficient Language Modelscited by 99
  11. 2019Risk-Averse Trust Region Optimization for Reward-Volatility Reductioncited by 53
  12. 2022Multi-Scale Adaptive Graph Neural Network for Multivariate Time Series Forecastingcited by 42
  13. 2022Preformer: Predictive Transformer with Multi-Scale Segment-wise Correlations for Long-Term Time Series Forecastingcited by 23
  14. 2020Multi-scale Transformer Language Modelscited by 12
  15. 2021Yformer: U-Net Inspired Transformer Architecture for Far Horizon Time Series Forecastingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 14; 2024: 17; 2025: 15; 2026: 8.

54 papers.

Sort:

By area:

  1. 2024TimeMixer: Decomposable Multiscale Mixing for Time Series Forecastingcited by 333
  2. 2024Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecastingcited by 102
  3. 2023Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospectscited by 97
  4. 2024FreDF: Learning to Forecast in the Frequency Domaincited by 77
  5. 2023Warpformer: A Multi-scale Modeling Approach for Irregular Clinical Time Seriescited by 54
  6. 2024Medformer: A Multi-Granularity Patching Transformer for Medical Time-Series Classificationcited by 53
  7. 2023A Survey on Time-Series Pre-Trained Modelscited by 52
  8. 2023Multi-resolution Time-Series Transformer for Long-term Forecastingcited by 41
  9. 2025HiMAE: Hierarchical Masked Autoencoders Discover Resolution-Specific Structure in Wearable Time Seriescited by 19
  10. 2023AutoCast++: Enhancing World Event Prediction with Zero-shot Ranking-based Context Retrievalcited by 18
  11. 2025Out-of-Distribution Generalization in Time Series: A Surveycited by 15
  12. 2023HUTFormer: Hierarchical U-Net Transformer for Long-Term Traffic Forecastingcited by 12
38 more
  1. 2024MSHyper: Multi-Scale Hypergraph Transformer for Long-Range Time Series Forecastingcited by 12
  2. 2023Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mappingcited by 11
  3. 2023A Multi-Scale Decomposition MLP-Mixer for Time Series Analysiscited by 10
  4. 2024Minusformer: Improving Time Series Forecasting by Progressively Learning Residualscited by 10
  5. 2024MCformer: Multivariate Time Series Forecasting with Mixed-Channels Transformercited by 10
  6. 2024Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecastingcited by 9
  7. 2025AttenGluco: Multimodal Transformer-Based Blood Glucose Forecasting on AI-READI Datasetcited by 9
  8. 2026The Forecast After the Forecast: A Post-Processing Shift in Time Seriescited by 9
  9. 2024Quantformer: from attention to profit with a quantitative transformer trading strategycited by 6
  10. 2025S4M: S4 for multivariate time series forecasting with Missing valuescited by 6
  11. 2023A Joint Time-frequency Domain Transformer for Multivariate Time Series Forecastingcited by 5
  12. 2024Survey and Taxonomy: The Role of Data-Centric AI in Transformer-Based Time Series Forecastingcited by 3
  13. 2023What Constitutes Good Contrastive Learning in Time-Series Forecasting?cited by 2
  14. 2024Machine Learning in Short-Reach Optical Systems: A Comprehensive Surveycited by 1
  15. 2024TwinS: Revisiting Non-Stationarity in Multivariate Time Series Forecastingcited by 1
  16. 2024Multiple-Resolution Tokenization for Time Series Forecasting with an Application to Pricingcited by 1
  17. 2025Spatially-Enhanced Temporal Fusion Transformer: Interpretable Multi-Output Prediction for Parametric Dynamical Systems with Time-Varying Inputscited by 1
  18. 2025STM3: Mixture of Multiscale Mamba for Long-Term Spatio-Temporal Time-Series Predictioncited by 1
  19. 2025CoSupFormer : A Contrastive Supervised learning approach for EEG signal Classificationcited by 1
  20. 2023Robust Reinforcement Learning Objectives for Sequential Recommender Systemscited by 0
  21. 2023Prompting-based Temporal Domain Generalizationcited by 0
  22. 2023MultiResFormer: Transformer with Adaptive Multi-Resolution Modeling for General Time Series Forecastingcited by 0
  23. 2023Take an Irregular Route: Enhance the Decoder of Time-Series Forecasting Transformercited by 0
  24. 2024Boosting MLPs with a Coarsening Strategy for Long-Term Time Series Forecastingcited by 0
  25. 2024Higher-order Cross-structural Embedding Model for Time Series Analysiscited by 0
  26. 2024A Wave is Worth 100 Words: Investigating Cross-Domain Transferability in Time Seriescited by 0
  27. 2024MuSiCNet: A Gradual Coarse-to-Fine Framework for Irregularly Sampled Multivariate Time Series Analysiscited by 0
  28. 2025MS-DFTVNet:A Long-Term Time Series Prediction Method Based on Multi-Scale Deformable Convolutioncited by 0
  29. 2025The Power of Architecture: Deep Dive into Transformer Architectures for Long-Term Time Series Forecastingcited by 0
  30. 2025IEFS-GMB: Gradient Memory Bank-Guided Feature Selection Based on Information Entropy for EEG Classification of Neurological Disorderscited by 0
  31. 2025GluMind: Multimodal Parallel Attention and Knowledge Retention for Robust Cross-Population Blood Glucose Forecastingcited by 0
  32. 2025DSAT-HD: Dual-Stream Adaptive Transformer with Hybrid Decomposition for Multivariate Time Series Forecastingcited by 0
  33. 2025InvDec: Inverted Decoder for Multivariate Time Series Forecasting with Separated Temporal and Variate Modelingcited by 0
  34. 2025Denoising Refinement Diffusion Models for Simultaneous Generation of Multi-scale Mobile Network Trafficcited by 0
  35. 2025Cisco Time Series Model Technical Reportcited by 0
  36. 2026PatchFormer: A Patch-Based Time Series Foundation Model with Hierarchical Masked Reconstruction and Cross-Domain Transfer Learning for Zero-Shot Multi-Horizon Forecastingcited by 0
  37. 2026Learning Recursive Multi-Scale Representations for Irregular Multivariate Time Series Forecastingcited by 0
  38. 2026CollideNet: Hierarchical Multi-scale Video Representation Learning with Disentanglement for Time-To-Collision Forecastingcited by 0

50 most cited of 54 All 54 (JSON)