# Contextualization Distillation from Large Language Model for Knowledge Graph Completion

arXiv: https://arxiv.org/abs/2402.01729 · HTML: https://mlsanity.com/paper/2402.01729/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 60 references that are in the map.
- First posted on arXiv 28 Jan 2024.

## Builds on (most cited first, 30 of 42)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://mlsanity.com/paper/2305.02301/) (2023)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020)
- [Composition-based Multi-Relational Graph Convolutional Networks](https://mlsanity.com/paper/1911.03082/) (2019)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [Teaching Small Language Models to Reason](https://mlsanity.com/paper/2212.08410/) (2022)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [HuatuoGPT, towards Taming Language Model to Be a Doctor](https://mlsanity.com/paper/2305.15075/) (2023)
- [SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization](https://mlsanity.com/paper/2212.10465/) (2022)
- [Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?](https://mlsanity.com/paper/2308.10168/) (2023)
- [Distilling Reasoning Capabilities into Smaller Language Models](https://mlsanity.com/paper/2212.00193/) (2022)
- [Learning Sequence Encoders for Temporal Knowledge Graph Completion](https://mlsanity.com/paper/1809.03202/) (2018)
- [ProsocialDialog: A Prosocial Backbone for Conversational Agents](https://mlsanity.com/paper/2205.12688/) (2022)
- [CoLAKE: Contextualized Language and Knowledge Embedding](https://mlsanity.com/paper/2010.00309/) (2020)
- [AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation](https://mlsanity.com/paper/2202.13047/) (2022)
- [One-Shot Relational Learning for Knowledge Graphs](https://mlsanity.com/paper/1808.09040/) (2018)

## Built on by (most cited first, 18 of 18)

- [Preference Leakage: A Contamination Problem in LLM-as-a-judge](https://mlsanity.com/paper/2502.01534/) (2025)
- [Exploring Large Language Models for Feature Selection: A Data-centric Perspective](https://mlsanity.com/paper/2408.12025/) (2024)
- [Learning from Diverse Reasoning Paths with Routing and Collaboration](https://mlsanity.com/paper/2508.16861/) (2025)
- [Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation](https://mlsanity.com/paper/2605.09253/) (2026)
- [Large Language Model Enhanced Knowledge Representation Learning: A Survey](https://mlsanity.com/paper/2407.00936/) (2024)
- [Large language models as oracles for instantiating ontologies with domain-specific knowledge](https://mlsanity.com/paper/2404.04108/) (2024)
- [A Contextualized BERT model for Knowledge Graph Completion](https://mlsanity.com/paper/2412.11016/) (2024)
- [Deploying Foundation Model Powered Agent Services: A Survey](https://mlsanity.com/paper/2412.13437/) (2024)
- [Leveraging Social Determinants of Health in Alzheimer's Research Using LLM-Augmented Literature Mining and Knowledge Graphs](https://mlsanity.com/paper/2410.09080/) (2024)
- [Is Large Language Model Good at Triple Set Prediction? An Empirical Study](https://mlsanity.com/paper/2412.18443/) (2024)
- [KG-BiLM: Knowledge Graph Embedding via Bidirectional Language Models](https://mlsanity.com/paper/2506.03576/) (2025)
- [OMNIA: Closing the Loop by Leveraging LLMs for Knowledge Graph Completion](https://mlsanity.com/paper/2603.11820/) (2026)
- [Beyond Entity Alignment: Towards Complete Knowledge Graph Alignment via Entity-Relation Synergy](https://mlsanity.com/paper/2407.17745/) (2024)
- [An Interdisciplinary and Cross-Task Review on Missing Data Imputation](https://mlsanity.com/paper/2511.01196/) (2025)
- [SASA: Semantic-Aware Contrastive Learning Framework with Separated Attention for Triple Classification](https://mlsanity.com/paper/2601.13035/) (2026)
- [RADAR: Reasoning as Discrimination with Aligned Representations for LLM-based Knowledge Graph Reasoning](https://mlsanity.com/paper/2602.21951/) (2026)
- [Conditional Diffusion Guided Knowledge Transfer for Multi-Domain Knowledge Graph Completion](https://mlsanity.com/paper/2607.03154/) (2026)
- [Probabilistic Concept-Aware Steering for Trustworthy LLM Inference](https://mlsanity.com/paper/2607.18259/) (2026)

## Abstract (arXiv)

While textual information significantly enhances the performance of pre-trained language models (PLMs) in knowledge graph completion (KGC), the static and noisy nature of existing corpora collected from Wikipedia articles or synsets definitions often limits the potential of PLM-based KGC models. To surmount these challenges, we introduce the Contextualization Distillation strategy, a versatile plug-in-and-play approach compatible with both discriminative and generative KGC frameworks. Our method begins by instructing large language models (LLMs) to transform compact, structural triplets into context-rich segments. Subsequently, we introduce two tailored auxiliary tasks, reconstruction and contextualization, allowing smaller KGC models to assimilate insights from these enriched triplets. Comprehensive evaluations across diverse datasets and KGC techniques highlight the efficacy and adaptability of our approach, revealing consistent performance enhancements irrespective of underlying pipelines or architectures. Moreover, our analysis makes our method more explainable and provides insight into generating path selection, as well as the choosing of suitable distillation tasks. All the code and data in this work will be released at https://github.com/David-Li0406/Contextulization-Distillation