# Towards Continual Knowledge Graph Embedding via Incremental Distillation

arXiv: https://arxiv.org/abs/2405.04453 · HTML: https://mlsanity.com/paper/2405.04453/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 26 references that are in the map.
- First posted on arXiv 7 May 2024.

## Builds on (most cited first, 13 of 13)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [SimplE Embedding for Link Prediction in Knowledge Graphs](https://mlsanity.com/paper/1802.04868/) (2018)
- [Knowledge Transfer for Out-of-Knowledge-Base Entities: A Graph Neural Network Approach](https://mlsanity.com/paper/1706.05674/) (2017)
- [Sentence Embedding Alignment for Lifelong Relation Extraction](https://mlsanity.com/paper/1903.02588/) (2019)
- [Continual Learning of Knowledge Graph Embeddings](https://mlsanity.com/paper/2101.05850/) (2021)
- [Lifelong Embedding Learning and Transfer for Growing Knowledge Graphs](https://mlsanity.com/paper/2211.15845/) (2022)
- [Open Question Answering with Weakly Supervised Embedding Models](https://mlsanity.com/paper/1404.4326/) (2014)
- [Disentangle-based Continual Graph Representation Learning](https://mlsanity.com/paper/2010.02565/) (2020)
- [DualDE: Dually Distilling Knowledge Graph Embedding for Faster and Cheaper Reasoning](https://mlsanity.com/paper/2009.05912/) (2020)

## Built on by (most cited first, 17 of 17)

- [Lifelong Learning of Large Language Model based Agents: A Roadmap](https://mlsanity.com/paper/2501.07278/) (2025)
- [Fast and Continual Knowledge Graph Embedding via Incremental LoRA](https://mlsanity.com/paper/2407.05705/) (2024)
- [Neurosymbolic Methods for Dynamic Knowledge Graphs](https://mlsanity.com/paper/2409.04572/) (2024)
- [Learning to Evolve: Bayesian-Guided Continual Knowledge Graph Embedding](https://mlsanity.com/paper/2508.02426/) (2025)
- [When Modalities Remember: Continual Learning for Multimodal Knowledge Graphs](https://mlsanity.com/paper/2604.02778/) (2026)
- [ETT-CKGE: Efficient Task-driven Tokens for Continual Knowledge Graph Embedding](https://mlsanity.com/paper/2506.08158/) (2025)
- [Croppable Knowledge Graph Embedding](https://mlsanity.com/paper/2407.02779/) (2024)
- [Capturing and Anticipating User Intents in Data Analytics via Knowledge Graphs](https://mlsanity.com/paper/2411.01023/) (2024)
- [Unlearning of Knowledge Graph Embedding via Preference Optimization](https://mlsanity.com/paper/2507.20566/) (2025)
- [SAGE: Scale-Aware Gradual Evolution for Continual Knowledge Graph Embedding](https://mlsanity.com/paper/2508.11347/) (2025)
- [Improving Continual Learning of Knowledge Graph Embeddings via Informed Initialization](https://mlsanity.com/paper/2511.11118/) (2025)
- [Revisiting Catastrophic Forgetting in Continual Knowledge Graph Embedding](https://mlsanity.com/paper/2604.19401/) (2026)
- [CMKL: Modality-Aware Continual Learning for Evolving Biomedical Knowledge Graphs](https://mlsanity.com/paper/2605.10510/) (2026)
- [PrimeKG-CL: A Continual Graph Learning Benchmark on Evolving Biomedical Knowledge Graphs](https://mlsanity.com/paper/2605.10529/) (2026)
- [MA-DAR: Manifold-Aligned Dynamic Adaptive Routing for Continual Temporal Knowledge Graph Reasoning](https://mlsanity.com/paper/2607.21949/) (2026)
- [Matched Excess-Outranker Regularization for Candidate-Set Interference in Continual Knowledge Graph Embedding](https://mlsanity.com/paper/2608.24273/) (2026)
- [When Successful Knowledge Graph Edits Displace Correct Answers: Rank-Level Locality beyond Parameter Support](https://mlsanity.com/paper/2609.12116/) (2026)

## Abstract (arXiv)

Traditional knowledge graph embedding (KGE) methods typically require preserving the entire knowledge graph (KG) with significant training costs when new knowledge emerges. To address this issue, the continual knowledge graph embedding (CKGE) task has been proposed to train the KGE model by learning emerging knowledge efficiently while simultaneously preserving decent old knowledge. However, the explicit graph structure in KGs, which is critical for the above goal, has been heavily ignored by existing CKGE methods. On the one hand, existing methods usually learn new triples in a random order, destroying the inner structure of new KGs. On the other hand, old triples are preserved with equal priority, failing to alleviate catastrophic forgetting effectively. In this paper, we propose a competitive method for CKGE based on incremental distillation (IncDE), which considers the full use of the explicit graph structure in KGs. First, to optimize the learning order, we introduce a hierarchical strategy, ranking new triples for layer-by-layer learning. By employing the inter- and intra-hierarchical orders together, new triples are grouped into layers based on the graph structure features. Secondly, to preserve the old knowledge effectively, we devise a novel incremental distillation mechanism, which facilitates the seamless transfer of entity representations from the previous layer to the next one, promoting old knowledge preservation. Finally, we adopt a two-stage training paradigm to avoid the over-corruption of old knowledge influenced by under-trained new knowledge. Experimental results demonstrate the superiority of IncDE over state-of-the-art baselines. Notably, the incremental distillation mechanism contributes to improvements of 0.2%-6.5% in the mean reciprocal rank (MRR) score.