# Inductive Relation Prediction by BERT

arXiv: https://arxiv.org/abs/2103.07102 · HTML: https://mlsanity.com/paper/2103.07102/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 12 Mar 2021.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [TuckER: Tensor Factorization for Knowledge Graph Completion](https://mlsanity.com/paper/1901.09590/) (2019)
- [DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoning](https://mlsanity.com/paper/1707.06690/) (2017)
- [A Review of Relational Machine Learning for Knowledge Graphs](https://mlsanity.com/paper/1503.00759/) (2015)
- [Inductive Relation Prediction by Subgraph Reasoning](https://mlsanity.com/paper/1911.06962/) (2019)
- [DRUM: End-To-End Differentiable Rule Mining On Knowledge Graphs](https://mlsanity.com/paper/1911.00055/) (2019)
- [Multiple Instance Learning: A Survey of Problem Characteristics and Applications](https://mlsanity.com/paper/1612.03365/) (2016)
- [Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learning](https://mlsanity.com/paper/1711.05851/) (2017)
- [Knowledge Transfer for Out-of-Knowledge-Base Entities: A Graph Neural Network Approach](https://mlsanity.com/paper/1706.05674/) (2017)
- [Knowledge Graph Completion via Complex Tensor Factorization](https://mlsanity.com/paper/1702.06879/) (2017)
- [Communicative Message Passing for Inductive Relation Reasoning](https://mlsanity.com/paper/2012.08911/) (2020)
- [Logic Attention Based Neighborhood Aggregation for Inductive Knowledge Graph Embedding](https://mlsanity.com/paper/1811.01399/) (2018)
- [Topology-Aware Correlations Between Relations for Inductive Link Prediction in Knowledge Graphs](https://mlsanity.com/paper/2103.03642/) (2021)
- [Variational Knowledge Graph Reasoning](https://mlsanity.com/paper/1803.06581/) (2018)
- [Inductively Representing Out-of-Knowledge-Graph Entities by Optimal Estimation Under Translational Assumptions](https://mlsanity.com/paper/2009.12765/) (2020)

## Built on by (most cited first, 27 of 27)

- [A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multimodal](https://mlsanity.com/paper/2212.05767/) (2022)
- [SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models](https://mlsanity.com/paper/2203.02167/) (2022)
- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [InGram: Inductive Knowledge Graph Embedding via Relation Graphs](https://mlsanity.com/paper/2305.19987/) (2023)
- [Graph Reasoning for Question Answering with Triplet Retrieval](https://mlsanity.com/paper/2305.18742/) (2023)
- [Zero-shot and Few-shot Learning with Knowledge Graphs: A Comprehensive Survey](https://mlsanity.com/paper/2112.10006/) (2021)
- [Double Equivariance for Inductive Link Prediction for Both New Nodes and New Relation Types](https://mlsanity.com/paper/2302.01313/) (2023)
- [KGLM: Integrating Knowledge Graph Structure in Language Models for Link Prediction](https://mlsanity.com/paper/2211.02744/) (2022)
- [Generalizing to Unseen Elements: A Survey on Knowledge Extrapolation for Knowledge Graphs](https://mlsanity.com/paper/2302.01859/) (2023)
- [Knowledge Graph Embedding: An Overview](https://mlsanity.com/paper/2309.12501/) (2023)
- [On the Evolution of Knowledge Graphs: A Survey and Perspective](https://mlsanity.com/paper/2310.04835/) (2023)
- [Large Language Models Can Better Understand Knowledge Graphs Than We Thought](https://mlsanity.com/paper/2402.11541/) (2024)
- [Incorporating Structured Sentences with Time-enhanced BERT for Fully-inductive Temporal Relation Prediction](https://mlsanity.com/paper/2304.04717/) (2023)
- [Negative Sampling in Knowledge Graph Representation Learning: A Review](https://mlsanity.com/paper/2402.19195/) (2024)
- [A Survey On Few-shot Knowledge Graph Completion with Structural and Commonsense Knowledge](https://mlsanity.com/paper/2301.01172/) (2023)
- [Multi-Aspect Explainable Inductive Relation Prediction by Sentence Transformer](https://mlsanity.com/paper/2301.01664/) (2023)
- [Query Structure Modeling for Inductive Logical Reasoning Over Knowledge Graphs](https://mlsanity.com/paper/2305.13585/) (2023)
- [Relation-Aware Network with Attention-Based Loss for Few-Shot Knowledge Graph Completion](https://mlsanity.com/paper/2306.09519/) (2023)
- [Textual Entailment for Effective Triple Validation in Object Prediction](https://mlsanity.com/paper/2401.16293/) (2024)
- [Certifying Knowledge Comprehension in LLMs](https://mlsanity.com/paper/2402.15929/) (2024)
- [Knowledge Graph Completion using Structural and Textual Embeddings](https://mlsanity.com/paper/2404.16206/) (2024)
- [Predicting from a Different Perspective: A Re-ranking Model for Inductive Knowledge Graph Completion](https://mlsanity.com/paper/2405.16902/) (2024)
- [KG-CF: Knowledge Graph Completion with Context Filtering under the Guidance of Large Language Models](https://mlsanity.com/paper/2501.02711/) (2025)
- [Temporal Knowledge Graph Reasoning Triggered by Memories](https://mlsanity.com/paper/2110.08765/) (2021)
- [One Subgraph for All: Efficient Reasoning on Opening Subgraphs for Inductive Knowledge Graph Completion](https://mlsanity.com/paper/2404.15807/) (2024)
- [Ontological Relations from Word Embeddings](https://mlsanity.com/paper/2408.00444/) (2024)
- [Fast-and-Frugal Text-Graph Transformers are Effective Link Predictors](https://mlsanity.com/paper/2408.06778/) (2024)

## Abstract (arXiv)

Relation prediction in knowledge graphs is dominated by embedding based methods which mainly focus on the transductive setting. Unfortunately, they are not able to handle inductive learning where unseen entities and relations are present and cannot take advantage of prior knowledge. Furthermore, their inference process is not easily explainable. In this work, we propose an all-in-one solution, called BERTRL (BERT-based Relational Learning), which leverages pre-trained language model and fine-tunes it by taking relation instances and their possible reasoning paths as training samples. BERTRL outperforms the SOTAs in 15 out of 18 cases in both inductive and transductive settings. Meanwhile, it demonstrates strong generalization capability in few-shot learning and is explainable.