# zrLLM: Zero-Shot Relational Learning on Temporal Knowledge Graphs with Large Language Models

arXiv: https://arxiv.org/abs/2311.10112 · HTML: https://mlsanity.com/paper/2311.10112/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 55 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [TuckER: Tensor Factorization for Knowledge Graph Completion](https://mlsanity.com/paper/1901.09590/) (2019)
- [Quaternion Knowledge Graph Embeddings](https://mlsanity.com/paper/1904.10281/) (2019)
- [Temporal Knowledge Graph Reasoning Based on Evolutional Representation Learning](https://mlsanity.com/paper/2104.10353/) (2021)
- [Question Answering Over Temporal Knowledge Graphs](https://mlsanity.com/paper/2106.01515/) (2021)
- [Learning from History: Modeling Temporal Knowledge Graphs with Sequential Copy-Generation Networks](https://mlsanity.com/paper/2012.08492/) (2020)
- [TimeTraveler: Reinforcement Learning for Temporal Knowledge Graph Forecasting](https://mlsanity.com/paper/2109.04101/) (2021)
- [Temporal Knowledge Graph Forecasting with Neural ODE](https://mlsanity.com/paper/2101.05151/) (2021)
- [TLogic: Temporal Logical Rules for Explainable Link Forecasting on Temporal Knowledge Graphs](https://mlsanity.com/paper/2112.08025/) (2021)
- [Temporal Knowledge Graph Reasoning with Historical Contrastive Learning](https://mlsanity.com/paper/2211.10904/) (2022)
- [Search from History and Reason for Future: Two-stage Reasoning on Temporal Knowledge Graphs](https://mlsanity.com/paper/2106.00327/) (2021)
- [Temporal Knowledge Graph Forecasting Without Knowledge Using In-Context Learning](https://mlsanity.com/paper/2305.10613/) (2023)
- [GenTKG: Generative Forecasting on Temporal Knowledge Graph with Large Language Models](https://mlsanity.com/paper/2310.07793/) (2023)
- [Learning to Sample and Aggregate: Few-shot Reasoning over Temporal Knowledge Graphs](https://mlsanity.com/paper/2210.08654/) (2022)
- [One-shot Learning for Temporal Knowledge Graphs](https://mlsanity.com/paper/2010.12144/) (2020)
- [ForecastTKGQuestions: A Benchmark for Temporal Question Answering and Forecasting over Temporal Knowledge Graphs](https://mlsanity.com/paper/2208.06501/) (2022)
- [Incorporating Structured Sentences with Time-enhanced BERT for Fully-inductive Temporal Relation Prediction](https://mlsanity.com/paper/2304.04717/) (2023)
- [Few-Shot Inductive Learning on Temporal Knowledge Graphs using Concept-Aware Information](https://mlsanity.com/paper/2211.08169/) (2022)
- [ECOLA: Enhanced Temporal Knowledge Embeddings with Contextualized Language Representations](https://mlsanity.com/paper/2203.09590/) (2022)
- [Meta-Learning Based Knowledge Extrapolation for Temporal Knowledge Graph](https://mlsanity.com/paper/2302.05640/) (2023)

## Built on by (most cited first, 13 of 13)

- [TGB 2.0: A Benchmark for Learning on Temporal Knowledge Graphs and Heterogeneous Graphs](https://mlsanity.com/paper/2406.09639/) (2024)
- [A Survey on Temporal Knowledge Graph: Representation Learning and Applications](https://mlsanity.com/paper/2403.04782/) (2024)
- [Foundation Models for Spatio-Temporal Data Science: A Tutorial and Survey](https://mlsanity.com/paper/2503.13502/) (2025)
- [Chain-of-History Reasoning for Temporal Knowledge Graph Forecasting](https://mlsanity.com/paper/2402.14382/) (2024)
- [Large Language Models for Knowledge Graph Embedding: A Survey](https://mlsanity.com/paper/2501.07766/) (2025)
- [Are Large Language Models Good Temporal Graph Learners?](https://mlsanity.com/paper/2506.05393/) (2025)
- [Towards Foundation Model on Temporal Knowledge Graph Reasoning](https://mlsanity.com/paper/2506.06367/) (2025)
- [Ground-Truth Subgraphs for Better Training and Evaluation of Knowledge Graph Augmented LLMs](https://mlsanity.com/paper/2511.04473/) (2025)
- [Self-Exploring Language Models for Explainable Link Forecasting on Temporal Graphs via Reinforcement Learning](https://mlsanity.com/paper/2509.00975/) (2025)
- [CountTRuCoLa: Rule Learning for Interpretable Temporal Knowledge Graph Forecasting](https://mlsanity.com/paper/2509.09474/) (2025)
- [AdaTKG: Adaptive Memory for Temporal Knowledge Graph Reasoning](https://mlsanity.com/paper/2605.07121/) (2026)
- [FITTER: Vocabulary-Agnostic Cross-Domain Inference on Temporal Knowledge Graphs](https://mlsanity.com/paper/2608.10668/) (2026)
- [CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inference](https://mlsanity.com/paper/2609.36689/) (2026)

## Abstract (arXiv)

Modeling evolving knowledge over temporal knowledge graphs (TKGs) has become a heated topic. Various methods have been proposed to forecast links on TKGs. Most of them are embedding-based, where hidden representations are learned to represent knowledge graph (KG) entities and relations based on the observed graph contexts. Although these methods show strong performance on traditional TKG forecasting (TKGF) benchmarks, they face a strong challenge in modeling the unseen zero-shot relations that have no prior graph context. In this paper, we try to mitigate this problem as follows. We first input the text descriptions of KG relations into large language models (LLMs) for generating relation representations, and then introduce them into embedding-based TKGF methods. LLM-empowered representations can capture the semantic information in the relation descriptions. This makes the relations, whether seen or unseen, with similar semantic meanings stay close in the embedding space, enabling TKGF models to recognize zero-shot relations even without any observed graph context. Experimental results show that our approach helps TKGF models to achieve much better performance in forecasting the facts with previously unseen relations, while still maintaining their ability in link forecasting regarding seen relations.