# Leveraging 2D Information for Long-term Time Series Forecasting with Vanilla Transformers

arXiv: https://arxiv.org/abs/2405.13810 · HTML: https://mlsanity.com/paper/2405.13810/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling](https://mlsanity.com/paper/1803.01271/) (2018)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://mlsanity.com/paper/1704.04110/) (2017)
- [iTransformer: Inverted Transformers Are Effective for Time Series Forecasting](https://mlsanity.com/paper/2310.06625/) (2023)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [Time-LLM: Time Series Forecasting by Reprogramming Large Language Models](https://mlsanity.com/paper/2310.01728/) (2023)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting](https://mlsanity.com/paper/1907.00235/) (2019)
- [Deep learning for time series classification: a review](https://mlsanity.com/paper/1809.04356/) (2018)
- [Transformers in Time Series: A Survey](https://mlsanity.com/paper/2202.07125/) (2022)
- [SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction](https://mlsanity.com/paper/2106.09305/) (2021)
- [TSMixer: An All-MLP Architecture for Time Series Forecasting](https://mlsanity.com/paper/2303.06053/) (2023)
- [Think Globally, Act Locally: A Deep Neural Network Approach to High-Dimensional Time Series Forecasting](https://mlsanity.com/paper/1905.03806/) (2019)
- [ETSformer: Exponential Smoothing Transformers for Time-series Forecasting](https://mlsanity.com/paper/2202.01381/) (2022)
- [Conditional Time Series Forecasting with Convolutional Neural Networks](https://mlsanity.com/paper/1703.04691/) (2017)
- [Lift Yourself Up: Retrieval-augmented Text Generation with Self Memory](https://mlsanity.com/paper/2305.02437/) (2023)
- [VMSMO: Learning to Generate Multimodal Summary for Video-based News Articles](https://mlsanity.com/paper/2010.05406/) (2020)
- [Long-term Forecasting using Higher Order Tensor RNNs](https://mlsanity.com/paper/1711.00073/) (2017)
- [Interactive Causal Structure Discovery in Earth System Sciences](https://mlsanity.com/paper/2107.01126/) (2021)
- [Follow the Timeline! Generating Abstractive and Extractive Timeline Summary in Chronological Order](https://mlsanity.com/paper/2301.00867/) (2023)

## Built on by (most cited first, 2 of 2)

- [Towards Robust Real-World Multivariate Time Series Forecasting: A Unified Framework for Dependency, Asynchrony, and Missingness](https://mlsanity.com/paper/2506.08660/) (2025)
- [GNA: Granular Neighbor Assembly for Retrieval-Augmented Multivariate Time-Series Forecasting](https://mlsanity.com/paper/2609.36281/) (2026)

## Abstract (arXiv)

Time series prediction is crucial for understanding and forecasting complex dynamics in various domains, ranging from finance and economics to climate and healthcare. Based on Transformer architecture, one approach involves encoding multiple variables from the same timestamp into a single temporal token to model global dependencies. In contrast, another approach embeds the time points of individual series into separate variate tokens. The former method faces challenges in learning variate-centric representations, while the latter risks missing essential temporal information critical for accurate forecasting. In our work, we introduce GridTST, a model that combines the benefits of two approaches using innovative multi-directional attentions based on a vanilla Transformer. We regard the input time series data as a grid, where the x-axis represents the time steps and the y-axis represents the variates. A vertical slicing of this grid combines the variates at each time step into a textittime token, while a horizontal slicing embeds the individual series across all time steps into a textitvariate token. Correspondingly, a textithorizontal attention mechanism focuses on time tokens to comprehend the correlations between data at various time steps, while a textitvertical, variate-aware textitattention is employed to grasp multivariate correlations. This combination enables efficient processing of information across both time and variate dimensions, thereby enhancing the model's analytical strength. % We also integrate the patch technique, segmenting time tokens into subseries-level patches, ensuring that local semantic information is retained in the embedding. The GridTST model consistently delivers state-of-the-art performance across various real-world datasets.