mlsanityData to 14 Oct 2026

Convergence Theory of Knowledge Distillation in Asynchronous P2P Gossip Learning Network

arXiv 2609.01952 ↗First posted 1 Sep 2026cs.LG cs.AIcited by 0

On the map: Knowledge distillation → Federated, robust and private distillation

Abstract (arXiv)

Decentralized, serverless learning increasingly connects devices running different architectures, where the standard tool, decentralized SGD, is undefined as models with different parameter counts cannot be averaged. Knowledge distillation (KD) exchanges soft predictions rather than weights and sidesteps this obstacle, yet convergence theory for fully decentralized, asynchronous peer-to-peer (P2P) KD is lacking. We provide one, relocating consensus from parameter space to function (output) space: a KD event is a geometric contraction operator in logit space on the peers' predictive distributions, which we analyse in the Hilbert space of predictions on a reference measure. Under standard smoothness/variance assumptions and two realizability assumptions, one bridging parameter SGD to the functional step and one controlling restricted task/KD alignment, the time-averaged functional stationarity and function-space disagreement converge at rate O(1/(η T)) to an O(η)+O(Bf2)+O(zetaf2) neighbourhood. Here Bf is the distance from the task optimum to the peers' reachable classes and zetaf measures persistent local-task heterogeneity. Across homogeneous, width-heterogeneous, and mixed-family networks of the experiments, KD contracts function disagreement by 40-61×, while isolated training does not. The sampled stationarity diagnostic has late transient exponents 0.99-1.90 on the shared-skeleton main runs, and the four-point step-size sweep exhibits the predicted transient: neighbourhood tradeoff.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Federated, robust and private distillation, one of 12 directions in Knowledge distillation. Builds on 22 of its 37 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

22 of 37 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022FedDKD: Federated Learning with Decentralized Knowledge Distillation2 May
  2. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning17 Mar
  3. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learning20 May

Other references · 19

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  7. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  8. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  9. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  10. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  11. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 367
  12. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
7 more
  1. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 93
  2. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  3. 2020Federated Mutual Learningcited by 71
  4. 2024Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchorscited by 21
  5. 2021Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Spacecited by 17
  6. 2022Data-heterogeneity-aware Mixing for Decentralized Learningcited by 12
  7. 2019Distributed Optimization Based on Gradient-tracking Revisited: Enhancing Convergence Rate via Surrogationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learning
  2. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillation
  3. 2026Randomly Initialized Networks Can Learn from Peer-to-Peer Consensus
  4. 2026Recursive Meta-Distillation: An Axiomatic Framework for Iterative Knowledge Refinement
  5. 2024Practical Insights into Knowledge Distillation for Pre-Trained Models
  6. 2026CALM: Class-wise Agreement and Label-gated Disagreement Modulation for Decentralized Federated Learning
  7. 2023Homogenizing Non-IID datasets via In-Distribution Knowledge Distillation for Decentralized Learning
  8. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions