# Incorporating Structured Sentences with Time-enhanced BERT for Fully-inductive Temporal Relation Prediction

arXiv: https://arxiv.org/abs/2304.04717 · HTML: https://mlsanity.com/paper/2304.04717/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 49 references that are in the map.
- First posted on arXiv 10 Apr 2023.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [Inductive Relation Prediction by Subgraph Reasoning](https://mlsanity.com/paper/1911.06962/) (2019)
- [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://mlsanity.com/paper/1907.12412/) (2019)
- [Time2Vec: Learning a Vector Representation of Time](https://mlsanity.com/paper/1907.05321/) (2019)
- [Learning Sequence Encoders for Temporal Knowledge Graph Completion](https://mlsanity.com/paper/1809.03202/) (2018)
- [Tensor Decompositions for temporal knowledge base completion](https://mlsanity.com/paper/2004.04926/) (2020)
- [Diachronic Embedding for Temporal Knowledge Graph Completion](https://mlsanity.com/paper/1907.03143/) (2019)
- [SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models](https://mlsanity.com/paper/2203.02167/) (2022)
- [CoLAKE: Contextualized Language and Knowledge Embedding](https://mlsanity.com/paper/2010.00309/) (2020)
- [TLogic: Temporal Logical Rules for Explainable Link Forecasting on Temporal Knowledge Graphs](https://mlsanity.com/paper/2112.08025/) (2021)
- [Communicative Message Passing for Inductive Relation Reasoning](https://mlsanity.com/paper/2012.08911/) (2020)
- [TeRo: A Time-aware Knowledge Graph Embedding via Temporal Rotation](https://mlsanity.com/paper/2010.01029/) (2020)
- [Time Masking for Temporal Language Models](https://mlsanity.com/paper/2110.06366/) (2021)
- [Entity-Duet Neural Ranking: Understanding the Role of Knowledge Graph Semantics in Neural Information Retrieval](https://mlsanity.com/paper/1805.07591/) (2018)
- [Open Domain Question Answering with A Unified Knowledge Interface](https://mlsanity.com/paper/2110.08417/) (2021)
- [Inductive Relation Prediction by BERT](https://mlsanity.com/paper/2103.07102/) (2021)
- [Time-aware Graph Neural Networks for Entity Alignment between Temporal Knowledge Graphs](https://mlsanity.com/paper/2203.02150/) (2022)
- [Language Models as Knowledge Embeddings](https://mlsanity.com/paper/2206.12617/) (2022)
- [Meta-Learning Based Knowledge Extrapolation for Temporal Knowledge Graph](https://mlsanity.com/paper/2302.05640/) (2023)
- [Geometric Algebra based Embeddings for Static and Temporal Knowledge Graph Completion](https://mlsanity.com/paper/2202.09464/) (2022)

## Built on by (most cited first, 9 of 9)

- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [YAGO 4.5: A Large and Clean Knowledge Base with a Rich Taxonomy](https://mlsanity.com/paper/2308.11884/) (2023)
- [zrLLM: Zero-Shot Relational Learning on Temporal Knowledge Graphs with Large Language Models](https://mlsanity.com/paper/2311.10112/) (2023)
- [LLM-Powered Explanations: Unraveling Recommendations Through Subgraph Reasoning](https://mlsanity.com/paper/2406.15859/) (2024)
- [Graphusion: Leveraging Large Language Models for Scientific Knowledge Graph Fusion and Construction in NLP Education](https://mlsanity.com/paper/2407.10794/) (2024)
- [Leveraging Large Language Models for Concept Graph Recovery and Question Answering in NLP Education](https://mlsanity.com/paper/2402.14293/) (2024)
- [Towards Foundation Model on Temporal Knowledge Graph Reasoning](https://mlsanity.com/paper/2506.06367/) (2025)
- [Deja vu: Contrastive Historical Modeling with Prefix-tuning for Temporal Knowledge Graph Reasoning](https://mlsanity.com/paper/2404.00051/) (2024)
- [Leveraging Pre-trained Language Models for Time Interval Prediction in Text-Enhanced Temporal Knowledge Graphs](https://mlsanity.com/paper/2309.16357/) (2023)

## Abstract (arXiv)

Temporal relation prediction in incomplete temporal knowledge graphs (TKGs) is a popular temporal knowledge graph completion (TKGC) problem in both transductive and inductive settings. Traditional embedding-based TKGC models (TKGE) rely on structured connections and can only handle a fixed set of entities, i.e., the transductive setting. In the inductive setting where test TKGs contain emerging entities, the latest methods are based on symbolic rules or pre-trained language models (PLMs). However, they suffer from being inflexible and not time-specific, respectively. In this work, we extend the fully-inductive setting, where entities in the training and test sets are totally disjoint, into TKGs and take a further step towards a more flexible and time-sensitive temporal relation prediction approach SST-BERT, incorporating Structured Sentences with Time-enhanced BERT. Our model can obtain the entity history and implicitly learn rules in the semantic space by encoding structured sentences, solving the problem of inflexibility. We propose to use a time masking MLM task to pre-train BERT in a corpus rich in temporal tokens specially generated for TKGs, enhancing the time sensitivity of SST-BERT. To compute the probability of occurrence of a target quadruple, we aggregate all its structured sentences from both temporal and semantic perspectives into a score. Experiments on the transductive datasets and newly generated fully-inductive benchmarks show that SST-BERT successfully improves over state-of-the-art baselines.