Knowledge-enhanced Transformer for Multivariate Long Sequence Time-series Forecasting
Not on the map. Why?
Abstract (arXiv)
Multivariate Long Sequence Time-series Forecasting (LSTF) has been a critical task across various real-world applications. Recent advancements focus on the application of transformer architectures attributable to their ability to capture temporal patterns effectively over extended periods. However, these approaches often overlook the inherent relationships and interactions between the input variables that could be drawn from their characteristic properties. In this paper, we aim to bridge this gap by integrating information-rich Knowledge Graph Embeddings (KGE) with state-of-the-art transformer-based architectures. We introduce a novel approach that encapsulates conceptual relationships among variables within a well-defined knowledge graph, forming dynamic and learnable KGEs for seamless integration into the transformer architecture. We investigate the influence of this integration into seminal architectures such as PatchTST, Autoformer, Informer, and Vanilla Transformer. Furthermore, we thoroughly investigate the performance of these knowledge-enhanced architectures along with their original implementations for long forecasting horizons and demonstrate significant improvement in the benchmark results. This enhancement empowers transformer-based architectures to address the inherent structural relation between variables. Our knowledge-enhanced approach improves the accuracy of multivariate LSTF by capturing complex temporal and relational dynamics across multiple domains. To substantiate the validity of our model, we conduct comprehensive experiments using Weather and Electric Transformer Temperature (ETT) datasets.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 59 references that are in the map. First posted on arXiv 17 Nov 2024.
Builds on
22 of 59 references.
References · 22
- 2017Attention Is All You Needcited by 60,361
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
- 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
- 2021Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecastingcited by 1,676
- 2022Are Transformers Effective for Time Series Forecasting?cited by 1,424
- 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,210
- 2017Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecastingcited by 1,136
- 2017Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networkscited by 879
- 2020A Generalization of Transformer Networks to Graphscited by 661
- 2020Time Series Forecasting With Deep Learning: A Surveycited by 459
10 more
- 2022Non-stationary Transformers: Exploring the Stationarity in Time Series Forecastingcited by 391
- 2021Spectral Temporal Graph Neural Network for Multivariate Time-series Forecastingcited by 299
- 2022Multivariate Time Series Forecasting with Dynamic Graph Neural ODEscited by 72
- 2021CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddingscited by 47
- 2021Long-Range Transformers for Dynamic Spatiotemporal Forecastingcited by 46
- 2023Improving Position Encoding of Transformers for Multivariate Time Series Classificationcited by 45
- 2022Multi-Scale Adaptive Graph Neural Network for Multivariate Time Series Forecastingcited by 42
- 2022Long-term Spatio-temporal Forecasting via Dynamic Multiple-Graph Attentioncited by 15
- 2022Relphormer: Relational Graph Transformer for Knowledge Graph Representationscited by 7
- 2022An Attention Free Long Short-Term Memory for Time Series Forecastingcited by 3
Built on by
Not cited yet.