Practical Insights into Knowledge Distillation for Pre-Trained Models
Not on the map. Why?
Abstract (arXiv)
This research investigates the enhancement of knowledge distillation (KD) processes in pre-trained models, an emerging field in knowledge transfer with significant implications for distributed training and federated learning environments. These environments benefit from reduced communication demands and accommodate various model architectures. Despite the adoption of numerous KD approaches for transferring knowledge among pre-trained models, a comprehensive understanding of KD's application in these scenarios is lacking. Our study conducts an extensive comparison of multiple KD techniques, including standard KD, tuned KD (via optimized temperature and weight parameters), deep mutual learning, and data partitioning KD. We assess these methods across various data distribution strategies to identify the most effective contexts for each. Through detailed examination of hyperparameter tuning, informed by extensive grid search evaluations, we pinpoint when adjustments are crucial to enhance model performance. This paper sheds light on optimal hyperparameter settings for distinct data partitioning scenarios and investigates KD's role in improving federated learning by minimizing communication rounds and expediting the training process. By filling a notable void in current research, our findings serve as a practical framework for leveraging KD in pre-trained models within collaborative and federated learning frameworks.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 19 of its 29 references that are in the map. First posted on arXiv 22 Feb 2024.
Builds on
19 of 29 references.
References · 19
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
- 2017Deep Mutual Learningcited by 621
- 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
- 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
- 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
- 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
- 2021Towards Understanding Knowledge Distillationcited by 158
- 2018Large scale distributed neural network training through online distillationcited by 144
- 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
- 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
- 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 108
7 more
- 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
- 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 91
- 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 67
- 2021Decentralized Federated Learning through Proxy Model Sharingcited by 56
- 2021FedGEMS: Federated Learning of Larger Server Models via Selective Knowledge Fusioncited by 49
- 2020Decentralized Federated Learning via Mutual Knowledge Transfercited by 42
- 2023Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Modelcited by 19
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.