# Confidence-aware Self-Semantic Distillation on Knowledge Graph Embedding

arXiv: https://arxiv.org/abs/2206.02963 · HTML: https://mlsanity.com/paper/2206.02963/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation.
- Builds on 26 of its 56 references that are in the map.
- First posted on arXiv 7 Jun 2022.

## Placed in

- Knowledge distillation / [Graph neural network distillation](https://mlsanity.com/distill/graph-neural-network-distillation/)

## Builds on (most cited first, 26 of 26)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Convolutional 2D Knowledge Graph Embeddings](https://mlsanity.com/paper/1707.01476/) (2017)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [TuckER: Tensor Factorization for Knowledge Graph Completion](https://mlsanity.com/paper/1901.09590/) (2019)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [Cross-Layer Distillation with Semantic Calibration](https://mlsanity.com/paper/2012.03236/) (2020)
- [Knowledge Distillation with the Reused Teacher Classifier](https://mlsanity.com/paper/2203.14001/) (2022)
- [Self-Knowledge Distillation with Progressive Refinement of Targets](https://mlsanity.com/paper/2006.12000/) (2020)
- [Snapshot Distillation: Teacher-Student Optimization in One Generation](https://mlsanity.com/paper/1812.00123/) (2018)
- [Unifying Knowledge Graph Learning and Recommendation: Towards a Better Understanding of User Preferences](https://mlsanity.com/paper/1902.06236/) (2019)
- [Question Answering with Subgraph Embeddings](https://mlsanity.com/paper/1406.3676/) (2014)
- [InteractE: Improving Convolution-based Knowledge Graph Embeddings by Increasing Feature Interactions](https://mlsanity.com/paper/1911.00219/) (2019)
- [Hypernetwork Knowledge Graph Embeddings](https://mlsanity.com/paper/1808.07018/) (2018)
- [LowFER: Low-rank Bilinear Pooling for Link Prediction](https://mlsanity.com/paper/2008.10858/) (2020)
- [DisenKGAT: Knowledge Graph Embedding with Disentangled Graph Attention Network](https://mlsanity.com/paper/2108.09628/) (2021)
- [DualDE: Dually Distilling Knowledge Graph Embedding for Faster and Cheaper Reasoning](https://mlsanity.com/paper/2009.05912/) (2020)
- [Knowledge Graph Embedding with Atrous Convolution and Residual Learning](https://mlsanity.com/paper/2010.12121/) (2020)
- [Learning with Retrospection](https://mlsanity.com/paper/2012.13098/) (2020)

## Abstract (arXiv)

Knowledge Graph Embedding (KGE), which projects entities and relations into continuous vector spaces, has garnered significant attention. Although high-dimensional KGE methods offer better performance, they come at the expense of significant computation and memory overheads. Decreasing embedding dimensions significantly deteriorates model performance. While several recent efforts utilize knowledge distillation or non-Euclidean representation learning to augment the effectiveness of low-dimensional KGE, they either necessitate a pre-trained high-dimensional teacher model or involve complex non-Euclidean operations, thereby incurring considerable additional computational costs. To address this, this work proposes Confidence-aware Self-Knowledge Distillation (CSD) that learns from the model itself to enhance KGE in a low-dimensional space. Specifically, CSD extracts knowledge from embeddings in previous iterations, which would be utilized to supervise the learning of the model in the next iterations. Moreover, a specific semantic module is developed to filter reliable knowledge by estimating the confidence of previously learned embeddings. This straightforward strategy bypasses the need for time-consuming pre-training of teacher models and can be integrated into various KGE methods to improve their performance. Our comprehensive experiments on six KGE backbones and four datasets underscore the effectiveness of the proposed CSD.