# TEAFormers: TEnsor-Augmented Transformers for Multi-Dimensional Time Series Forecasting

arXiv: https://arxiv.org/abs/2410.20439 · HTML: https://mlsanity.com/paper/2410.20439/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 55 references that are in the map.
- First posted on arXiv 27 Oct 2024.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting](https://mlsanity.com/paper/1905.03806/) (2019)
- [ETSformer: Exponential Smoothing Transformers for Time-series Forecasting](https://mlsanity.com/paper/2202.01381/) (2022)
- [On the Expressive Power of Deep Learning: A Tensor Analysis](https://mlsanity.com/paper/1509.05009/) (2015)
- [Improving the Transformer Translation Model with Document-Level Context](https://mlsanity.com/paper/1810.03581/) (2018)
- [DSformer: A Double Sampling Transformer for Multivariate Time Series Long-term Prediction](https://mlsanity.com/paper/2308.03274/) (2023)
- [Long-Range Transformers for Dynamic Spatiotemporal Forecasting](https://mlsanity.com/paper/2109.12218/) (2021)
- [Tensor Regression Networks](https://mlsanity.com/paper/1707.08308/) (2017)
- [When Do Contrastive Learning Signals Help Spatio-Temporal Graph Forecasting?](https://mlsanity.com/paper/2108.11873/) (2021)
- [Multilinear tensor regression for longitudinal relational data](https://mlsanity.com/paper/1412.0048/) (2014)
- [A GNN-RNN Approach for Harnessing Geospatial and Temporal Information: Application to Crop Yield Prediction](https://mlsanity.com/paper/2111.08900/) (2021)
- [Factor Models for High-Dimensional Tensor Time Series](https://mlsanity.com/paper/1905.07530/) (2019)
- [Autoregressive Models for Matrix-Valued Time Series](https://mlsanity.com/paper/1812.08916/) (2018)
- [Tensor Contraction Layers for Parsimonious Deep Nets](https://mlsanity.com/paper/1706.00439/) (2017)
- [Constrained Factor Models for High-Dimensional Matrix-Variate Time Series](https://mlsanity.com/paper/1710.06075/) (2017)
- [Tensor-view Topological Graph Neural Network](https://mlsanity.com/paper/2401.12007/) (2024)
- [Statistical Inference for High-Dimensional Matrix-Variate Factor Model](https://mlsanity.com/paper/2001.01890/) (2020)
- [CP Factor Model for Dynamic Tensors](https://mlsanity.com/paper/2110.15517/) (2021)
- [Graph Tensor Networks: An Intuitive Framework for Designing Large-Scale Neural Learning Systems on Multiple Domains](https://mlsanity.com/paper/2303.13565/) (2023)
- [Factor Models for Matrix-Valued High-Dimensional Time Series](https://mlsanity.com/paper/1610.01889/) (2016)
- [Tensor Factor Model Estimation by Iterative Projection](https://mlsanity.com/paper/2006.02611/) (2020)
- [Semi-parametric TEnsor Factor Analysis by Iteratively Projected Singular Value Decomposition](https://mlsanity.com/paper/2007.02404/) (2020)
- [Rank Determination in Tensor Factor Model](https://mlsanity.com/paper/2011.07131/) (2020)
- [Modeling Dynamic Transport Network with Matrix Factor Models: with an Application to International Trade Flow](https://mlsanity.com/paper/1901.00769/) (2019)
- [Modeling Multivariate Spatial-Temporal Data with Latent Low-Dimensional Dynamics](https://mlsanity.com/paper/2002.01305/) (2020)
- [Tensor Principal Component Analysis in High Dimensional CP Models](https://mlsanity.com/paper/2108.04428/) (2021)

## Built on by (most cited first, 1 of 1)

- [Dual-Channel Tensor Neural Networks: Finite-Sample Theory and Conformal Structure Selection](https://mlsanity.com/paper/2605.19122/) (2026)

## Abstract (arXiv)

Multi-dimensional time series data, such as matrix and tensor-variate time series, are increasingly prevalent in fields such as economics, finance, and climate science. Traditional Transformer models, though adept with sequential data, do not effectively preserve these multi-dimensional structures, as their internal operations in effect flatten multi-dimensional observations into vectors, thereby losing critical multi-dimensional relationships and patterns. To address this, we introduce the Tensor-Augmented Transformer (TEAFormer), a novel method that incorporates tensor expansion and compression within the Transformer framework to maintain and leverage the inherent multi-dimensional structures, thus reducing computational costs and improving prediction accuracy. The core feature of the TEAFormer, the Tensor-Augmentation (TEA) module, utilizes tensor expansion to enhance multi-view feature learning and tensor compression for efficient information aggregation and reduced computational load. The TEA module is not just a specific model architecture but a versatile component that is highly compatible with the attention mechanism and the encoder-decoder structure of Transformers, making it adaptable to existing Transformer architectures. Our comprehensive experiments, which integrate the TEA module into three popular time series Transformer models across three real-world benchmarks, show significant performance enhancements, highlighting the potential of TEAFormers for cutting-edge time series forecasting.