Low-Dimensional Federated Knowledge Graph Embedding via Knowledge Distillation
Not on the map. Why?
Abstract (arXiv)
Federated Knowledge Graph Embedding (FKGE) aims to facilitate collaborative learning of entity and relation embeddings from distributed Knowledge Graphs (KGs) across multiple clients, while preserving data privacy. Training FKGE models with higher dimensions is typically favored due to their potential for achieving superior performance. However, high-dimensional embeddings present significant challenges in terms of storage resource and inference speed. Unlike traditional KG embedding methods, FKGE involves multiple client-server communication rounds, where communication efficiency is critical. Existing embedding compression methods for traditional KGs may not be directly applicable to FKGE as they often require multiple model trainings which potentially incur substantial communication costs. In this paper, we propose a light-weight component based on Knowledge Distillation (KD) which is titled FedKD and tailored specifically for FKGE methods. During client-side local training, FedKD facilitates the low-dimensional student model to mimic the score distribution of triples from the high-dimensional teacher model using KL divergence loss. Unlike traditional KD way, FedKD adaptively learns a temperature to scale the score of positive triples and separately adjusts the scores of corresponding negative triples using a predefined temperature, thereby mitigating teacher over-confidence issue. Furthermore, we dynamically adjust the weight of KD loss to optimize the training process. Extensive experiments on three datasets support the effectiveness of FedKD.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 11 Aug 2024.
Builds on
25 of 37 references.
References · 25
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2022Generative Adversarial Networkscited by 5,603
- 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
- 2016Complex Embeddings for Simple Link Predictioncited by 897
- 2019RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Spacecited by 893
- 2021Model-Contrastive Federated Learningcited by 762
- 2020Personalized Federated Learning with Moreau Envelopescited by 612
- 2017Word Translation Without Parallel Datacited by 480
- 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 371
- 2020Federated learning with hierarchical clustering of local updates to improve training on non-IID datacited by 222
13 more
- 2021Personalized Federated Learning using Hypernetworkscited by 210
- 2019Reinforcement Knowledge Graph Reasoning for Explainable Recommendationcited by 102
- 2023From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labelscited by 69
- 2020FedE: Embedding Knowledge Graphs in Federated Settingcited by 46
- 2021Differentially Private Federated Knowledge Graphs Embeddingcited by 45
- 2023Heterogeneous Federated Knowledge Graph Embedding Learning and Unlearningcited by 39
- 2022Asymmetric Temperature Scaling Makes Larger Networks Teach Well Againcited by 23
- 2022Efficient Federated Learning on Knowledge Graphs via Privacy-preserving Relation Embedding Aggregationcited by 18
- 2020DualDE: Dually Distilling Knowledge Graph Embedding for Faster and Cheaper Reasoningcited by 17
- 2022Boosting Graph Neural Networks via Adaptive Knowledge Distillationcited by 14
- 2024Beyond Similarity: Personalized Federated Recommendation with Composite Aggregationcited by 4
- 2024Personalized Federated Knowledge Graph Embedding with Client-Wise Relation Graphcited by 3
- 2024Communication-Efficient Federated Knowledge Graph Embedding with Entity-Wise Top-K Sparsificationcited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.