mlsanityData to 7 Oct 2026

Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model Fusion

arXiv 2208.07978 ↗First posted 16 Aug 2022cs.DC cs.CR cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Concerned with user data privacy, this paper presents a new federated learning (FL) method that trains machine learning models on edge devices without accessing sensitive data. Traditional FL methods, although privacy-protective, fail to manage model heterogeneity and incur high communication costs due to their reliance on aggregation methods. To address this limitation, we propose a resource-aware FL method that aggregates local knowledge from edge models and distills it into robust global knowledge through knowledge distillation. This method allows efficient multi-model knowledge fusion and the deployment of resource-aware models while preserving model heterogeneity. Our method improves communication cost and performance in heterogeneous data and models compared to existing FL algorithms. Notably, it reduces the communication cost of ResNet-32 by up to 50% and VGG-11 by up to 10× while delivering superior performance.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 33 of its 49 references that are in the map. First posted on arXiv 16 Aug 2022.

Builds on

33 of 49 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  8. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  9. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  10. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  11. 2017Deep Mutual Learningcited by 621
  12. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
21 more
  1. 2019Similarity-Preserving Knowledge Distillationcited by 507
  2. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  3. 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 347
  4. 2021Ensemble deep learning: A reviewcited by 283
  5. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  6. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
  7. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  8. 2019Privacy-preserving Federated Brain Tumour Segmentationcited by 169
  9. 2020Survey of Personalization Techniques for Federated Learningcited by 144
  10. 2020Federated Accelerated Stochastic Gradient Descentcited by 85
  11. 2021Federated Evaluation and Tuning for On-Device Personalization: System Design & Applicationscited by 84
  12. 2020Turbo-Aggregate: Breaking the Quadratic Aggregation Barrier in Secure Federated Learningcited by 78
  13. 2021OpenFL: An open-source framework for Federated Learningcited by 58
  14. 2019Federated Learning Of Out-Of-Vocabulary Wordscited by 51
  15. 2020Federated Knowledge Distillationcited by 51
  16. 2019Energy Demand Prediction with Federated Learning for Electric Vehicle Networkscited by 46
  17. 2019Ensemble Knowledge Distillation for Learning Improved and Efficient Networkscited by 46
  18. 2021Topology-Aware Network Pruning using Multi-stage Graph Embedding and Reinforcement Learningcited by 30
  19. 2020Auto Graph Encoder-Decoder for Neural Network Pruningcited by 26
  20. 2020Secure Aggregation with Heterogeneous Quantization in Federated Learningcited by 21
  21. 2021Heterogeneous Federated Learning using Dynamic Model Pruning and Adaptive Gradientcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 7; 2025: 1.

10 papers.

  1. 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 49
  2. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learningcited by 21
  3. 2024Federated Model Heterogeneous Matryoshka Representation Learningcited by 21
  4. 2024Federated Distillation: A Surveycited by 14
  5. 2024Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directionscited by 12
  6. 2024Training Machine Learning models at the Edge: A Surveycited by 11
  7. 2023Knowledge Distillation in Federated Edge Learning: A Surveycited by 5
  8. 2024pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in Heterogeneous Federated Learningcited by 3
  9. 2025Bridging Generalization Gap of Heterogeneous Federated Clients Using Generative Modelscited by 1
  10. 2024Federated Learning with Workload Reduction through Partial Training of Client Models and Entropy-Based Data Selectioncited by 0