# Multi-scale Transformer Pyramid Networks for Multivariate Time Series Forecasting

arXiv: https://arxiv.org/abs/2308.11946 · HTML: https://mlsanity.com/paper/2308.11946/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 23 Aug 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction](https://mlsanity.com/paper/2106.09305/) (2021)
- [Temporal Pattern Attention for Multivariate Time Series Forecasting](https://mlsanity.com/paper/1809.04206/) (2018)

## Built on by (most cited first, 9 of 9)

- [From Similarity to Superiority: Channel Clustering for Time Series Forecasting](https://mlsanity.com/paper/2404.01340/) (2024)
- [TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model](https://mlsanity.com/paper/2409.02322/) (2024)
- [ms-Mamba: Multi-scale Mamba for Time-Series Forecasting](https://mlsanity.com/paper/2504.07654/) (2025)
- [Sparse Transformer with Local and Seasonal Adaptation for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2312.06874/) (2023)
- [Toward Physics-guided Time Series Embedding](https://mlsanity.com/paper/2410.06651/) (2024)
- [Routing Channel-Patch Dependencies in Time Series Forecasting with Graph Spectral Decomposition](https://mlsanity.com/paper/2603.13702/) (2026)
- [TwinS: Revisiting Non-Stationarity in Multivariate Time Series Forecasting](https://mlsanity.com/paper/2406.03710/) (2024)
- [A Supervised Contrastive Learning Pretrain-Finetune Approach for Time Series](https://mlsanity.com/paper/2311.12290/) (2023)
- [Extreme Adaptive Transformer for Time Series Forecasting](https://mlsanity.com/paper/2607.02437/) (2026)

## Abstract (arXiv)

Multivariate Time Series (MTS) forecasting involves modeling temporal dependencies within historical records. Transformers have demonstrated remarkable performance in MTS forecasting due to their capability to capture long-term dependencies. However, prior work has been confined to modeling temporal dependencies at either a fixed scale or multiple scales that exponentially increase (most with base 2). This limitation hinders their effectiveness in capturing diverse seasonalities, such as hourly and daily patterns. In this paper, we introduce a dimension invariant embedding technique that captures short-term temporal dependencies and projects MTS data into a higher-dimensional space, while preserving the dimensions of time steps and variables in MTS data. Furthermore, we present a novel Multi-scale Transformer Pyramid Network (MTPNet), specifically designed to effectively capture temporal dependencies at multiple unconstrained scales. The predictions are inferred from multi-scale latent representations obtained from transformers at various scales. Extensive experiments on nine benchmark datasets demonstrate that the proposed MTPNet outperforms recent state-of-the-art methods.