# Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting

arXiv: https://arxiv.org/abs/2206.04038 · HTML: https://mlsanity.com/paper/2206.04038/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 50 references that are in the map.
- First posted on arXiv 8 Jun 2022.

## Builds on (most cited first, 27 of 27)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [A General and Adaptive Robust Loss Function](https://mlsanity.com/paper/1701.03077/) (2017)
- [Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding](https://mlsanity.com/paper/2103.15358/) (2021)
- [Multi-level Wavelet-CNN for Image Restoration](https://mlsanity.com/paper/1805.07071/) (2018)
- [FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecasting](https://mlsanity.com/paper/2205.08897/) (2022)
- [N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting](https://mlsanity.com/paper/2201.12886/) (2022)
- [Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case](https://mlsanity.com/paper/2001.08317/) (2020)
- [Hierarchical Multiscale Recurrent Neural Networks](https://mlsanity.com/paper/1609.01704/) (2016)
- [Hierarchical Transformers Are More Efficient Language Models](https://mlsanity.com/paper/2110.13711/) (2021)
- [Multi-Scale Convolutional Neural Networks for Time Series Classification](https://mlsanity.com/paper/1603.06995/) (2016)
- [Risk-Averse Trust Region Optimization for Reward-Volatility Reduction](https://mlsanity.com/paper/1912.03193/) (2019)
- [Multi-Scale Adaptive Graph Neural Network for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2201.04828/) (2022)
- [Preformer: Predictive Transformer with Multi-Scale Segment-wise Correlations for Long-Term Time Series Forecasting](https://mlsanity.com/paper/2202.11356/) (2022)
- [Multi-scale Transformer Language Models](https://mlsanity.com/paper/2005.00581/) (2020)
- [Yformer: U-Net Inspired Transformer Architecture for Far Horizon Time Series Forecasting](https://mlsanity.com/paper/2110.08255/) (2021)

## Built on by (most cited first, 30 of 54)

- [TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting](https://mlsanity.com/paper/2405.14616/) (2024)
- [Pathformer: Multi-scale Transformers with Adaptive Pathways for Time Series Forecasting](https://mlsanity.com/paper/2402.05956/) (2024)
- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)
- [FreDF: Learning to Forecast in the Frequency Domain](https://mlsanity.com/paper/2402.02399/) (2024)
- [Warpformer: A Multi-scale Modeling Approach for Irregular Clinical Time Series](https://mlsanity.com/paper/2306.09368/) (2023)
- [Medformer: A Multi-Granularity Patching Transformer for Medical Time-Series Classification](https://mlsanity.com/paper/2405.19363/) (2024)
- [A Survey on Time-Series Pre-Trained Models](https://mlsanity.com/paper/2305.10716/) (2023)
- [Multi-resolution Time-Series Transformer for Long-term Forecasting](https://mlsanity.com/paper/2311.04147/) (2023)
- [HiMAE: Hierarchical Masked Autoencoders Discover Resolution-Specific Structure in Wearable Time Series](https://mlsanity.com/paper/2510.25785/) (2025)
- [AutoCast++: Enhancing World Event Prediction with Zero-shot Ranking-based Context Retrieval](https://mlsanity.com/paper/2310.01880/) (2023)
- [Out-of-Distribution Generalization in Time Series: A Survey](https://mlsanity.com/paper/2503.13868/) (2025)
- [HUTFormer: Hierarchical U-Net Transformer for Long-Term Traffic Forecasting](https://mlsanity.com/paper/2307.14596/) (2023)
- [MSHyper: Multi-Scale Hypergraph Transformer for Long-Range Time Series Forecasting](https://mlsanity.com/paper/2401.09261/) (2024)
- [Revisiting Long-term Time Series Forecasting: An Investigation on Linear Mapping](https://mlsanity.com/paper/2305.10721/) (2023)
- [A Multi-Scale Decomposition MLP-Mixer for Time Series Analysis](https://mlsanity.com/paper/2310.11959/) (2023)
- [Minusformer: Improving Time Series Forecasting by Progressively Learning Residuals](https://mlsanity.com/paper/2402.02332/) (2024)
- [MCformer: Multivariate Time Series Forecasting with Mixed-Channels Transformer](https://mlsanity.com/paper/2403.09223/) (2024)
- [The Forecast After the Forecast: A Post-Processing Shift in Time Series](https://mlsanity.com/paper/2601.20280/) (2026)
- [Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting](https://mlsanity.com/paper/2405.16312/) (2024)
- [AttenGluco: Multimodal Transformer-Based Blood Glucose Forecasting on AI-READI Dataset](https://mlsanity.com/paper/2502.09919/) (2025)
- [Quantformer: from attention to profit with a quantitative transformer trading strategy](https://mlsanity.com/paper/2404.00424/) (2024)
- [S4M: S4 for multivariate time series forecasting with Missing values](https://mlsanity.com/paper/2503.00900/) (2025)
- [A Joint Time-frequency Domain Transformer for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2305.14649/) (2023)
- [Survey and Taxonomy: The Role of Data-Centric AI in Transformer-Based Time Series Forecasting](https://mlsanity.com/paper/2407.19784/) (2024)
- [What Constitutes Good Contrastive Learning in Time-Series Forecasting?](https://mlsanity.com/paper/2306.12086/) (2023)
- [Machine Learning in Short-Reach Optical Systems: A Comprehensive Survey](https://mlsanity.com/paper/2405.09557/) (2024)
- [TwinS: Revisiting Non-Stationarity in Multivariate Time Series Forecasting](https://mlsanity.com/paper/2406.03710/) (2024)
- [Multiple-Resolution Tokenization for Time Series Forecasting with an Application to Pricing](https://mlsanity.com/paper/2407.03185/) (2024)
- [Spatially-Enhanced Temporal Fusion Transformer: Interpretable Multi-Output Prediction for Parametric Dynamical Systems with Time-Varying Inputs](https://mlsanity.com/paper/2505.00473/) (2025)
- [STM3: Mixture of Multiscale Mamba for Long-Term Spatio-Temporal Time-Series Prediction](https://mlsanity.com/paper/2508.12247/) (2025)

## Abstract (arXiv)

The performance of time series forecasting has recently been greatly improved by the introduction of transformers. In this paper, we propose a general multi-scale framework that can be applied to the state-of-the-art transformer-based time series forecasting models (FEDformer, Autoformer, etc.). By iteratively refining a forecasted time series at multiple scales with shared weights, introducing architecture adaptations, and a specially-designed normalization scheme, we are able to achieve significant performance improvements, from 5.5% to 38.5% across datasets and transformer architectures, with minimal additional computational overhead. Via detailed ablation studies, we demonstrate the effectiveness of each of our contributions across the architecture and methodology. Furthermore, our experiments on various public datasets demonstrate that the proposed improvements outperform their corresponding baseline counterparts. Our code is publicly available in https://github.com/BorealisAI/scaleformer.