A Kernel Perspective on Distillation-based Collaborative Learning
Not on the map. Why?
Abstract (arXiv)
Over the past decade, there is a growing interest in collaborative learning that can enhance AI models of multiple parties. However, it is still challenging to enhance performance them without sharing private data and models from individual parties. One recent promising approach is to develop distillation-based algorithms that exploit unlabeled public data but the results are still unsatisfactory in both theory and practice. To tackle this problem, we rigorously analyze a representative distillation-based algorithm in the view of kernel regression. This work provides the first theoretical results to prove the (nearly) minimax optimality of the nonparametric collaborative learning algorithm that does not directly share local data or models in massively distributed statistically heterogeneous environments. Inspired by our theoretical results, we also propose a practical distillation-based collaborative learning algorithm based on neural network architecture. Our algorithm successfully bridges the gap between our theoretical assumptions and practical settings with neural networks through feature kernel matching. We simulate various regression tasks to verify our theory and demonstrate the practical feasibility of our proposed algorithm.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 26 of its 72 references that are in the map. First posted on arXiv 23 Oct 2024.
Builds on
26 of 72 references.
References · 26
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2018How To Backdoor Federated Learningcited by 745
- 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
- 2020Personalized Federated Learning with Moreau Envelopescited by 612
- 2020Federated Learning with Matched Averagingcited by 564
- 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
- 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
- 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 178
14 more
- 2020Byzantine-Robust Learning on Heterogeneous Datasets via Bucketingcited by 125
- 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 108
- 2013Divide and Conquer Kernel Ridge Regression: A Distributed Algorithm with Minimax Optimal Ratescited by 64
- 2017Personalized and Private Peer-to-Peer Machine Learningcited by 54
- 2022Architecture Agnostic Federated Learning for Neural Networkscited by 30
- 2023On the Optimality of Misspecified Spectral Algorithmscited by 28
- 2020Distributed Kernel Ridge Regression with Communicationscited by 14
- 2021A Non-parametric View of FedAvg and FedProx: Beyond Stationary Pointscited by 13
- 2024On the Saturation Effect of Kernel Ridge Regressioncited by 12
- 2020Group Knowledge Transfer: Federated Learning of Large CNNs at the Edgecited by 10
- 2023Collaborative Learning via Prediction Consensuscited by 8
- 2020Kernel Interpolation of High Dimensional Scattered Datacited by 6
- 2021Test-time Collective Predictioncited by 2
- 2017Optimal Rates for Learning with Nystr"om Stochastic Gradient Methodscited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.