# Low-Dimensional Federated Knowledge Graph Embedding via Knowledge Distillation

arXiv: https://arxiv.org/abs/2408.05748 · HTML: https://mlsanity.com/paper/2408.05748/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 11 Aug 2024.

## Placed in

- Knowledge distillation / [Federated, robust and private distillation](https://mlsanity.com/distill/federated-robust-and-private-distillation/)

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space](https://mlsanity.com/paper/1902.10197/) (2019)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Personalized Federated Learning with Moreau Envelopes](https://mlsanity.com/paper/2006.08848/) (2020)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection](https://mlsanity.com/paper/1907.09693/) (2019)
- [Federated learning with hierarchical clustering of local updates to improve training on non-IID data](https://mlsanity.com/paper/2004.11791/) (2020)
- [Personalized Federated Learning using Hypernetworks](https://mlsanity.com/paper/2103.04628/) (2021)
- [Reinforcement Knowledge Graph Reasoning for Explainable Recommendation](https://mlsanity.com/paper/1906.05237/) (2019)
- [From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels](https://mlsanity.com/paper/2303.13005/) (2023)
- [FedE: Embedding Knowledge Graphs in Federated Setting](https://mlsanity.com/paper/2010.12882/) (2020)
- [Differentially Private Federated Knowledge Graphs Embedding](https://mlsanity.com/paper/2105.07615/) (2021)
- [Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearning](https://mlsanity.com/paper/2302.02069/) (2023)
- [Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again](https://mlsanity.com/paper/2210.04427/) (2022)
- [Efficient Federated Learning on Knowledge Graphs via Privacy-preserving Relation Embedding Aggregation](https://mlsanity.com/paper/2203.09553/) (2022)
- [DualDE: Dually Distilling Knowledge Graph Embedding for Faster and Cheaper Reasoning](https://mlsanity.com/paper/2009.05912/) (2020)
- [Boosting Graph Neural Networks via Adaptive Knowledge Distillation](https://mlsanity.com/paper/2210.05920/) (2022)
- [Beyond Similarity: Personalized Federated Recommendation with Composite Aggregation](https://mlsanity.com/paper/2406.03933/) (2024)
- [Personalized Federated Knowledge Graph Embedding with Client-Wise Relation Graph](https://mlsanity.com/paper/2406.11943/) (2024)
- [Communication-Efficient Federated Knowledge Graph Embedding with Entity-Wise Top-K Sparsification](https://mlsanity.com/paper/2406.13225/) (2024)

## Built on by (most cited first, 1 of 1)

- [Cost Characterization of Vertically Partitioned Federated Knowledge Graphs](https://mlsanity.com/paper/2609.13664/) (2026)

## Abstract (arXiv)

Federated Knowledge Graph Embedding (FKGE) aims to facilitate collaborative learning of entity and relation embeddings from distributed Knowledge Graphs (KGs) across multiple clients, while preserving data privacy. Training FKGE models with higher dimensions is typically favored due to their potential for achieving superior performance. However, high-dimensional embeddings present significant challenges in terms of storage resource and inference speed. Unlike traditional KG embedding methods, FKGE involves multiple client-server communication rounds, where communication efficiency is critical. Existing embedding compression methods for traditional KGs may not be directly applicable to FKGE as they often require multiple model trainings which potentially incur substantial communication costs. In this paper, we propose a light-weight component based on Knowledge Distillation (KD) which is titled FedKD and tailored specifically for FKGE methods. During client-side local training, FedKD facilitates the low-dimensional student model to mimic the score distribution of triples from the high-dimensional teacher model using KL divergence loss. Unlike traditional KD way, FedKD adaptively learns a temperature to scale the score of positive triples and separately adjusts the scores of corresponding negative triples using a predefined temperature, thereby mitigating teacher over-confidence issue. Furthermore, we dynamically adjust the weight of KD loss to optimize the training process. Extensive experiments on three datasets support the effectiveness of FedKD.