# Scalable Transformer for High Dimensional Multivariate Time Series Forecasting

arXiv: https://arxiv.org/abs/2408.04245 · HTML: https://mlsanity.com/paper/2408.04245/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 79 references that are in the map.
- First posted on arXiv 8 Aug 2024.

## Builds on (most cited first, 30 of 31)

- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [A Transformer-based Framework for Multivariate Time Series Representation Learning](https://mlsanity.com/paper/2010.02803/) (2020)
- [Spatial-Temporal Identity: A Simple yet Effective Baseline for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2208.05233/) (2022)
- [High-Dimensional Multivariate Forecasting with Low-Rank Gaussian Copula Processes](https://mlsanity.com/paper/1910.03002/) (2019)
- [TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2306.09364/) (2023)
- [FourierGNN: Rethinking Multivariate Time Series Forecasting from a Pure Graph Perspective](https://mlsanity.com/paper/2311.06190/) (2023)
- [Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2206.09113/) (2022)
- [Contrast Everything: A Hierarchical Contrastive Framework for Medical Time-Series](https://mlsanity.com/paper/2310.14017/) (2023)
- [Forecast Evaluation for Data Scientists: Common Pitfalls and Best Practices](https://mlsanity.com/paper/2203.10716/) (2022)
- [Warpformer: A Multi-scale Modeling Approach for Irregular Clinical Time Series](https://mlsanity.com/paper/2306.09368/) (2023)
- [DSformer: A Double Sampling Transformer for Multivariate Time Series Long-term Prediction](https://mlsanity.com/paper/2308.03274/) (2023)
- [TFAD: A Decomposition Time Series Anomaly Detection Architecture with Time-Frequency Analysis](https://mlsanity.com/paper/2210.09693/) (2022)
- [Domain Adversarial Spatial-Temporal Network: A Transferable Framework for Short-term Traffic Forecasting across Cities](https://mlsanity.com/paper/2202.03630/) (2022)
- [Learning the Evolutionary and Multi-scale Graph Structure for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2206.13816/) (2022)
- [Networked Time Series Imputation via Position-aware Graph Enhanced Variational Autoencoders](https://mlsanity.com/paper/2305.18612/) (2023)
- [Temporal and Heterogeneous Graph Neural Network for Financial Time Series Prediction](https://mlsanity.com/paper/2305.08740/) (2023)
- [HAGEN: Homophily-Aware Graph Convolutional Recurrent Network for Crime Forecasting](https://mlsanity.com/paper/2109.12846/) (2021)
- [LightCTS: A Lightweight Framework for Correlated Time Series Forecasting](https://mlsanity.com/paper/2302.11974/) (2023)
- [Adaptive Graph Spatial-Temporal Transformer Network for Traffic Flow Forecasting](https://mlsanity.com/paper/2207.05064/) (2022)
- [Towards Long-Term Time-Series Forecasting: Feature, Pattern, and Distribution](https://mlsanity.com/paper/2301.02068/) (2023)
- [Localised Adaptive Spatial-Temporal Graph Neural Network](https://mlsanity.com/paper/2306.06930/) (2023)
- [DuETA: Traffic Congestion Propagation Pattern Modeling via Efficient Graph Learning for ETA Prediction at Baidu Maps](https://mlsanity.com/paper/2208.06979/) (2022)
- [MemDA: Forecasting Urban Time Series with Memory-based Drift Adaptation](https://mlsanity.com/paper/2309.14216/) (2023)
- [Adaptive Dependency Learning Graph Neural Networks](https://mlsanity.com/paper/2312.03903/) (2023)
- [Fragment and Integrate Network (FIN): A Novel Spatial-Temporal Modeling Based on Long Sequential Behavior for Online Food Ordering Click-Through Rate Prediction](https://mlsanity.com/paper/2308.15703/) (2023)
- [Deep Graphs - a general framework to represent and analyze heterogeneous complex systems across scales](https://mlsanity.com/paper/1604.00971/) (2016)

## Built on by (most cited first, 3 of 3)

- [MoTime: A Dataset Suite for Multimodal Time Series Forecasting](https://mlsanity.com/paper/2505.15072/) (2025)
- [CITRAS: Covariate-Informed Transformer for Time Series Forecasting](https://mlsanity.com/paper/2503.24007/) (2025)
- [Unicorn: Scaling High-Dimensional Time Series Forecasting via Universal Correlation Modeling](https://mlsanity.com/paper/2605.30376/) (2026)

## Abstract (arXiv)

Deep models for Multivariate Time Series (MTS) forecasting have recently demonstrated significant success. Channel-dependent models capture complex dependencies that channel-independent models cannot capture. However, the number of channels in real-world applications outpaces the capabilities of existing channel-dependent models, and contrary to common expectations, some models underperform the channel-independent models in handling high-dimensional data, which raises questions about the performance of channel-dependent models. To address this, our study first investigates the reasons behind the suboptimal performance of these channel-dependent models on high-dimensional MTS data. Our analysis reveals that two primary issues lie in the introduced noise from unrelated series that increases the difficulty of capturing the crucial inter-channel dependencies, and challenges in training strategies due to high-dimensional data. To address these issues, we propose STHD, the Scalable Transformer for High-Dimensional Multivariate Time Series Forecasting. STHD has three components: a) Relation Matrix Sparsity that limits the noise introduced and alleviates the memory issue; b) ReIndex applied as a training strategy to enable a more flexible batch size setting and increase the diversity of training data; and c) Transformer that handles 2-D inputs and captures channel dependencies. These components jointly enable STHD to manage the high-dimensional MTS while maintaining computational feasibility. Furthermore, experimental results show STHD's considerable improvement on three high-dimensional datasets: Crime-Chicago, Wiki-People, and Traffic. The source code and dataset are publicly available https://github.com/xinzzzhou/ScalableTransformer4HighDimensionMTSF.git.