# Knowledge-enhanced Transformer for Multivariate Long Sequence Time-series Forecasting

arXiv: https://arxiv.org/abs/2411.11046 · HTML: https://mlsanity.com/paper/2411.11046/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 59 references that are in the map.
- First posted on arXiv 17 Nov 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting](https://mlsanity.com/paper/1709.04875/) (2017)
- [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://mlsanity.com/paper/1703.07015/) (2017)
- [A Generalization of Transformer Networks to Graphs](https://mlsanity.com/paper/2012.09699/) (2020)
- [Time Series Forecasting With Deep Learning: A Survey](https://mlsanity.com/paper/2004.13408/) (2020)
- [Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting](https://mlsanity.com/paper/2205.14415/) (2022)
- [Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting](https://mlsanity.com/paper/2103.07719/) (2021)
- [Multivariate Time Series Forecasting with Dynamic Graph Neural ODEs](https://mlsanity.com/paper/2202.08408/) (2022)
- [CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings](https://mlsanity.com/paper/2106.03143/) (2021)
- [Long-Range Transformers for Dynamic Spatiotemporal Forecasting](https://mlsanity.com/paper/2109.12218/) (2021)
- [Improving Position Encoding of Transformers for Multivariate Time Series Classification](https://mlsanity.com/paper/2305.16642/) (2023)
- [Multi-Scale Adaptive Graph Neural Network for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2201.04828/) (2022)
- [Long-term Spatio-temporal Forecasting via Dynamic Multiple-Graph Attention](https://mlsanity.com/paper/2204.11008/) (2022)
- [Relphormer: Relational Graph Transformer for Knowledge Graph Representations](https://mlsanity.com/paper/2205.10852/) (2022)
- [An Attention Free Long Short-Term Memory for Time Series Forecasting](https://mlsanity.com/paper/2209.09548/) (2022)

## Abstract (arXiv)

Multivariate Long Sequence Time-series Forecasting (LSTF) has been a critical task across various real-world applications. Recent advancements focus on the application of transformer architectures attributable to their ability to capture temporal patterns effectively over extended periods. However, these approaches often overlook the inherent relationships and interactions between the input variables that could be drawn from their characteristic properties. In this paper, we aim to bridge this gap by integrating information-rich Knowledge Graph Embeddings (KGE) with state-of-the-art transformer-based architectures. We introduce a novel approach that encapsulates conceptual relationships among variables within a well-defined knowledge graph, forming dynamic and learnable KGEs for seamless integration into the transformer architecture. We investigate the influence of this integration into seminal architectures such as PatchTST, Autoformer, Informer, and Vanilla Transformer. Furthermore, we thoroughly investigate the performance of these knowledge-enhanced architectures along with their original implementations for long forecasting horizons and demonstrate significant improvement in the benchmark results. This enhancement empowers transformer-based architectures to address the inherent structural relation between variables. Our knowledge-enhanced approach improves the accuracy of multivariate LSTF by capturing complex temporal and relational dynamics across multiple domains. To substantiate the validity of our model, we conduct comprehensive experiments using Weather and Electric Transformer Temperature (ETT) datasets.