# I Know What You Do Not Know: Knowledge Graph Embedding via Co-distillation Learning

arXiv: https://arxiv.org/abs/2208.09828 · HTML: https://mlsanity.com/paper/2208.09828/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 21 Aug 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [A Survey on Knowledge Graphs: Representation, Acquisition and Applications](https://mlsanity.com/paper/2002.00388/) (2020)
- [Composition-based Multi-Relational Graph Convolutional Networks](https://mlsanity.com/paper/1911.03082/) (2019)
- [Decoupled Knowledge Distillation](https://mlsanity.com/paper/2203.08679/) (2022)
- [KG-BERT: BERT for Knowledge Graph Completion](https://mlsanity.com/paper/1909.03193/) (2019)
- [TuckER: Tensor Factorization for Knowledge Graph Completion](https://mlsanity.com/paper/1901.09590/) (2019)
- [Low-Dimensional Hyperbolic Knowledge Graph Embeddings](https://mlsanity.com/paper/2005.00545/) (2020)
- [Knowledge Graph Embedding for Link Prediction: A Comparative Analysis](https://mlsanity.com/paper/2002.00819/) (2020)
- [Learning to Exploit Long-term Relational Dependencies in Knowledge Graphs](https://mlsanity.com/paper/1905.04914/) (2019)
- [Sequence-to-Sequence Knowledge Graph Completion and Question Answering](https://mlsanity.com/paper/2203.10321/) (2022)
- [Modeling Relation Paths for Representation Learning of Knowledge Bases](https://mlsanity.com/paper/1506.00379/) (2015)
- [HittER: Hierarchical Transformers for Knowledge Graph Embeddings](https://mlsanity.com/paper/2008.12813/) (2020)
- [CoKE: Contextualized Knowledge Graph Embedding](https://mlsanity.com/paper/1911.02168/) (2019)
- [Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical Study](https://mlsanity.com/paper/2106.09700/) (2021)
- [MLMLM: Link Prediction with Mean Likelihood Masked Language Model](https://mlsanity.com/paper/2009.07058/) (2020)
- [Convolutional Complex Knowledge Graph Embeddings](https://mlsanity.com/paper/2008.03130/) (2020)
- [Predicting Completeness in Knowledge Bases](https://mlsanity.com/paper/1612.05786/) (2016)
- [Reasoning Through Memorization: Nearest Neighbor Knowledge Graph Embeddings](https://mlsanity.com/paper/2201.05575/) (2022)

## Built on by (most cited first, 15 of 15)

- [HyperFormer: Enhancing Entity and Relation Interaction for Hyper-Relational Knowledge Graph Completion](https://mlsanity.com/paper/2308.06512/) (2023)
- [Finetuning Generative Large Language Models with Discrimination Instructions for Knowledge Graph Completion](https://mlsanity.com/paper/2407.16127/) (2024)
- [Inductive Knowledge Graph Completion with GNNs and Rules: An Analysis](https://mlsanity.com/paper/2308.07942/) (2023)
- [Knowledge Graph Error Detection with Contrastive Confidence Adaption](https://mlsanity.com/paper/2312.12108/) (2023)
- [Multi-view Contrastive Learning for Entity Typing over Knowledge Graphs](https://mlsanity.com/paper/2310.12008/) (2023)
- [HyperMono: A Monotonicity-aware Approach to Hyper-Relational Knowledge Representation](https://mlsanity.com/paper/2404.09848/) (2024)
- [GLTW: Joint Improved Graph Transformer and LLM via Three-Word Language for Knowledge Graph Completion](https://mlsanity.com/paper/2502.11471/) (2025)
- [DuetGraph: Coarse-to-Fine Knowledge Graph Reasoning with Dual-Pathway Global-Local Fusion](https://mlsanity.com/paper/2507.11229/) (2025)
- [On the Theoretical Limitations of Embedding-based Link Prediction](https://mlsanity.com/paper/2506.22271/) (2025)
- [COTET: Cross-view Optimal Transport for Knowledge Graph Entity Typing](https://mlsanity.com/paper/2405.13602/) (2024)
- [Deep Sparse Latent Feature Models for Knowledge Graph Completion](https://mlsanity.com/paper/2411.15694/) (2024)
- [DrKGC: Dynamic Subgraph Retrieval-Augmented LLMs for Knowledge Graph Completion across General and Biomedical Domains](https://mlsanity.com/paper/2506.00708/) (2025)
- [SLiNT: Structure-aware Language Model with Injection and Contrastive Training for Knowledge Graph Completion](https://mlsanity.com/paper/2509.06531/) (2025)
- [Tokenization, Fusion and Decoupling: Bridging the Granularity Mismatch Between Large Language Models and Knowledge Graphs](https://mlsanity.com/paper/2602.22698/) (2026)
- [GS-Quant: Granular Semantic and Generative Structural Quantization for Knowledge Graph Completion](https://mlsanity.com/paper/2604.21649/) (2026)

## Abstract (arXiv)

Knowledge graph (KG) embedding seeks to learn vector representations for entities and relations. Conventional models reason over graph structures, but they suffer from the issues of graph incompleteness and long-tail entities. Recent studies have used pre-trained language models to learn embeddings based on the textual information of entities and relations, but they cannot take advantage of graph structures. In the paper, we show empirically that these two kinds of features are complementary for KG embedding. To this end, we propose CoLE, a Co-distillation Learning method for KG Embedding that exploits the complementarity of graph structures and text information. Its graph embedding model employs Transformer to reconstruct the representation of an entity from its neighborhood subgraph. Its text embedding model uses a pre-trained language model to generate entity representations from the soft prompts of their names, descriptions, and relational neighbors. To let the two model promote each other, we propose co-distillation learning that allows them to distill selective knowledge from each other's prediction logits. In our co-distillation learning, each model serves as both a teacher and a student. Experiments on benchmark datasets demonstrate that the two models outperform their related baselines, and the ensemble method CoLE with co-distillation learning advances the state-of-the-art of KG embedding.