# KGLM: Integrating Knowledge Graph Structure in Language Models for Link Prediction

arXiv: https://arxiv.org/abs/2211.02744 · HTML: https://mlsanity.com/paper/2211.02744/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 44 references that are in the map.
- First posted on arXiv 4 Nov 2022.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [A Review of Relational Machine Learning for Knowledge Graphs](https://mlsanity.com/paper/1503.00759/) (2015)
- [Knowledge Graph Embedding for Link Prediction: A Comparative Analysis](https://mlsanity.com/paper/2002.00819/) (2020)
- [A Capsule Network-based Embedding Model for Knowledge Graph Completion and Search Personalization](https://mlsanity.com/paper/1808.04122/) (2018)
- [Inductive Relation Prediction by BERT](https://mlsanity.com/paper/2103.07102/) (2021)
- [LineaRE: Simple but Powerful Knowledge Graph Embedding for Link Prediction](https://mlsanity.com/paper/2004.10037/) (2020)
- [QuatDE: Dynamic Quaternion Embedding for Knowledge Graph Completion](https://mlsanity.com/paper/2105.09002/) (2021)

## Built on by (most cited first, 15 of 15)

- [Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey](https://mlsanity.com/paper/2311.07914/) (2023)
- [Large Language Models Hallucination: A Comprehensive Survey](https://mlsanity.com/paper/2510.06265/) (2025)
- [Exploring Large Language Models for Knowledge Graph Completion](https://mlsanity.com/paper/2308.13916/) (2023)
- [Knowledge Graph Large Language Model (KG-LLM) for Link Prediction](https://mlsanity.com/paper/2403.07311/) (2024)
- [CaLa: Complementary Association Learning for Augmenting Composed Image Retrieval](https://mlsanity.com/paper/2405.19149/) (2024)
- [MKGL: Mastery of a Three-Word Language](https://mlsanity.com/paper/2410.07526/) (2024)
- [SynerGPT: In-Context Learning for Personalized Drug Synergy Prediction and Drug Design](https://mlsanity.com/paper/2307.11694/) (2023)
- [Knowledge Graph-Infused Fine-Tuning for Structured Reasoning in Large Language Models](https://mlsanity.com/paper/2508.14427/) (2025)
- [K-ON: Stacking Knowledge On the Head Layer of Large Language Model](https://mlsanity.com/paper/2502.06257/) (2025)
- [Knowledge Graph Completion using Structural and Textual Embeddings](https://mlsanity.com/paper/2404.16206/) (2024)
- [Semi-Automated Construction of Food Composition Knowledge Base](https://mlsanity.com/paper/2301.11322/) (2023)
- [Progressive Distillation Based on Masked Generation Feature Method for Knowledge Graph Completion](https://mlsanity.com/paper/2401.12997/) (2024)
- [Are We Wasting Time? A Fast, Accurate Performance Evaluation Framework for Knowledge Graph Link Predictors](https://mlsanity.com/paper/2402.00053/) (2024)
- [DrKGC: Dynamic Subgraph Retrieval-Augmented LLMs for Knowledge Graph Completion across General and Biomedical Domains](https://mlsanity.com/paper/2506.00708/) (2025)
- [Learning Chain Of Thoughts Prompts for Predicting Entities, Relations, and even Literals on Knowledge Graphs](https://mlsanity.com/paper/2604.12651/) (2026)

## Abstract (arXiv)

The ability of knowledge graphs to represent complex relationships at scale has led to their adoption for various needs including knowledge representation, question-answering, and recommendation systems. Knowledge graphs are often incomplete in the information they represent, necessitating the need for knowledge graph completion tasks. Pre-trained and fine-tuned language models have shown promise in these tasks although these models ignore the intrinsic information encoded in the knowledge graph, namely the entity and relation types. In this work, we propose the Knowledge Graph Language Model (KGLM) architecture, where we introduce a new entity/relation embedding layer that learns to differentiate distinctive entity and relation types, therefore allowing the model to learn the structure of the knowledge graph. In this work, we show that further pre-training the language models with this additional embedding layer using the triples extracted from the knowledge graph, followed by the standard fine-tuning phase sets a new state-of-the-art performance for the link prediction task on the benchmark datasets.