# Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting--Full Version

arXiv: https://arxiv.org/abs/2204.13767 · HTML: https://mlsanity.com/paper/2204.13767/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 48 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Graph WaveNet for Deep Spatial-Temporal Graph Modeling](https://mlsanity.com/paper/1906.00121/) (2019)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks](https://mlsanity.com/paper/2005.11650/) (2020)
- [Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting](https://mlsanity.com/paper/2007.02842/) (2020)
- [Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting](https://mlsanity.com/paper/2103.07719/) (2021)
- [Structured Sequence Modeling with Graph Convolutional Recurrent Networks](https://mlsanity.com/paper/1612.07659/) (2016)
- [A Dual-Stage Attention-Based Recurrent Neural Network for Time Series Prediction](https://mlsanity.com/paper/1704.02971/) (2017)
- [Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing](https://mlsanity.com/paper/2006.03236/) (2020)
- [Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context](https://mlsanity.com/paper/1805.04623/) (2018)
- [Hierarchical Transformers for Long Document Classification](https://mlsanity.com/paper/1910.10781/) (2019)
- [Scalable Vision Transformers with Hierarchical Pooling](https://mlsanity.com/paper/2103.10619/) (2021)
- [Towards Spatio-Temporal Aware Traffic Time Series Forecasting--Full Version](https://mlsanity.com/paper/2203.15737/) (2022)
- [AutoCTS: Automated Correlated Time Series Forecasting -- Extended Version](https://mlsanity.com/paper/2112.11174/) (2021)
- [Unsupervised Path Representation Learning with Curriculum Negative Sampling](https://mlsanity.com/paper/2106.09373/) (2021)
- [Unsupervised Time Series Outlier Detection with Diversity-Driven Convolutional Ensembles -- Extended Version](https://mlsanity.com/paper/2111.11108/) (2021)
- [Graph Attention Recurrent Neural Networks for Correlated Time Series Forecasting -- Full version](https://mlsanity.com/paper/2103.10760/) (2021)
- [Seq-U-Net: A One-Dimensional Causal U-Net for Efficient Sequence Modelling](https://mlsanity.com/paper/1911.06393/) (2019)

## Built on by (most cited first, 30 of 59)

- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis](https://mlsanity.com/paper/2310.06119/) (2023)
- [Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects](https://mlsanity.com/paper/2306.10125/) (2023)
- [LightTS: Lightweight Time Series Classification with Adaptive Ensemble Distillation -- Extended Version](https://mlsanity.com/paper/2302.12721/) (2023)
- [DSformer: A Double Sampling Transformer for Multivariate Time Series Long-term Prediction](https://mlsanity.com/paper/2308.03274/) (2023)
- [A Survey of Deep Learning and Foundation Models for Time Series Forecasting](https://mlsanity.com/paper/2401.13912/) (2024)
- [LightPath: Lightweight and Scalable Path Representation Learning](https://mlsanity.com/paper/2307.10171/) (2023)
- [Joint Neural Architecture and Hyperparameter Search for Correlated Time Series Forecasting](https://mlsanity.com/paper/2211.16126/) (2022)
- [Parsimony or Capability? Decomposition Delivers Both in Long-term Time Series Forecasting](https://mlsanity.com/paper/2401.11929/) (2024)
- [SSD-TS: Exploring the Potential of Linear State Space Models for Diffusion Models in Time Series Imputation](https://mlsanity.com/paper/2410.13338/) (2024)
- [Dynamic Hypergraph Structure Learning for Traffic Flow Forecasting](https://mlsanity.com/paper/2309.12028/) (2023)
- [HUTFormer: Hierarchical U-Net Transformer for Long-Term Traffic Forecasting](https://mlsanity.com/paper/2307.14596/) (2023)
- [MSHyper: Multi-Scale Hypergraph Transformer for Long-Range Time Series Forecasting](https://mlsanity.com/paper/2401.09261/) (2024)
- [Fully Automated Correlated Time Series Forecasting in Minutes](https://mlsanity.com/paper/2411.05833/) (2024)
- [Diffusion Models for Time Series Forecasting: A Survey](https://mlsanity.com/paper/2507.14507/) (2025)
- [A Systematic Review for Transformer-based Long-term Series Forecasting](https://mlsanity.com/paper/2310.20218/) (2023)
- [Deep Learning for Time Series Forecasting: A Survey](https://mlsanity.com/paper/2503.10198/) (2025)
- [Text2TimeSeries: Enhancing Financial Forecasting through Time Series Prediction Updates with Event-Driven Insights from Large Language Models](https://mlsanity.com/paper/2407.03689/) (2024)
- [MixLinear: Extreme Low Resource Multivariate Time Series Forecasting with 0.1K Parameters](https://mlsanity.com/paper/2410.02081/) (2024)
- [GBT: Two-stage transformer framework for non-stationary time series forecasting](https://mlsanity.com/paper/2307.08302/) (2023)
- [SAGDFN: A Scalable Adaptive Graph Diffusion Forecasting Network for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2406.12282/) (2024)
- [Anomaly Detection and Generation with Diffusion Models: A Survey](https://mlsanity.com/paper/2506.09368/) (2025)
- [ASTGI: Adaptive Spatio-Temporal Graph Interactions for Irregular Multivariate Time Series Forecasting](https://mlsanity.com/paper/2509.23313/) (2025)
- [Efficient Time Series Processing for Transformers and State-Space Models through Token Merging](https://mlsanity.com/paper/2405.17951/) (2024)
- [When Will It Fail?: Anomaly to Prompt for Forecasting Future Anomalies in Time Series](https://mlsanity.com/paper/2506.23596/) (2025)
- [Quantformer: from attention to profit with a quantitative transformer trading strategy](https://mlsanity.com/paper/2404.00424/) (2024)
- [XXLTraffic: Expanding and Extremely Long Traffic forecasting beyond test adaptation](https://mlsanity.com/paper/2406.12693/) (2024)
- [ReFocus: Reinforcing Mid-Frequency and Key-Frequency Modeling for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2502.16890/) (2025)
- [ST-RAP: A Spatio-Temporal Framework for Real Estate Appraisal](https://mlsanity.com/paper/2308.10609/) (2023)

## Abstract (arXiv)

A variety of real-world applications rely on far future information to make decisions, thus calling for efficient and accurate long sequence multivariate time series forecasting. While recent attention-based forecasting models show strong abilities in capturing long-term dependencies, they still suffer from two key limitations. First, canonical self attention has a quadratic complexity w.r.t. the input time series length, thus falling short in efficiency. Second, different variables' time series often have distinct temporal dynamics, which existing studies fail to capture, as they use the same model parameter space, e.g., projection matrices, for all variables' time series, thus falling short in accuracy. To ensure high efficiency and accuracy, we propose Triformer, a triangular, variable-specific attention. (i) Linear complexity: we introduce a novel patch attention with linear complexity. When stacking multiple layers of the patch attentions, a triangular structure is proposed such that the layer sizes shrink exponentially, thus maintaining linear complexity. (ii) Variable-specific parameters: we propose a light-weight method to enable distinct sets of model parameters for different variables' time series to enhance accuracy without compromising efficiency and memory usage. Strong empirical evidence on four datasets from multiple domains justifies our design choices, and it demonstrates that Triformer outperforms state-of-the-art methods w.r.t. both accuracy and efficiency. This is an extended version of "Triformer: Triangular, Variable-Specific Attentions for Long Sequence Multivariate Time Series Forecasting", to appear in IJCAI 2022 [Cirstea et al., 2022a], including additional experimental results.