mlsanityData to 7 Oct 2026

DFML: Decentralized Federated Mutual Learning

arXiv 2402.01863 ↗First posted 2 Feb 2024cs.LG cs.AI cs.DCcited by 4

Not on the map. Why?

Abstract (arXiv)

In the realm of real-world devices, centralized servers in Federated Learning (FL) present challenges including communication bottlenecks and susceptibility to a single point of failure. Additionally, contemporary devices inherently exhibit model and data heterogeneity. Existing work lacks a Decentralized FL (DFL) framework capable of accommodating such heterogeneity without imposing architectural restrictions or assuming the availability of public data. To address these issues, we propose a Decentralized Federated Mutual Learning (DFML) framework that is serverless, supports nonrestrictive heterogeneous models, and avoids reliance on public data. DFML effectively handles model and data heterogeneity through mutual learning, which distills knowledge between clients, and cyclically varying the amount of supervision and distillation signals. Extensive experimental results demonstrate consistent effectiveness of DFML in both convergence speed and global accuracy, outperforming prevalent baselines under various conditions. For example, with the CIFAR-100 dataset and 50 clients, DFML achieves a substantial increase of +17.20% and +19.95% in global accuracy under Independent and Identically Distributed (IID) and non-IID data shifts, respectively.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 31 of its 38 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

31 of 38 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  10. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  11. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  12. 2017Deep Mutual Learningcited by 621
19 more
  1. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  2. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  3. 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 215
  4. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  5. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  6. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 162
  7. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
  8. 2022Decentralized Federated Learning: Fundamentals, State of the Art, Frameworks, Trends, and Challengescited by 137
  9. 2021Vision Transformer for Small-Size Datasetscited by 122
  10. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
  11. 2019BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learningcited by 115
  12. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  13. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  14. 2023Decentralized Federated Learning: A Survey and Perspectivecited by 86
  15. 2019BAM! Born-Again Multi-Task Networks for Natural Language Understandingcited by 85
  16. 2020Practical One-Shot Federated Learning for Cross-Silo Settingcited by 76
  17. 2020Federated Mutual Learningcited by 70
  18. 2020Decentralized Federated Learning via Mutual Knowledge Transfercited by 42
  19. 2023Re-Weighted Softmax Cross-Entropy to Control Forgetting in Federated Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025NoT: Federated Unlearning via Weight Negationcited by 15
  2. 2026CRAD: Class-wise Reliability-Aware Distillation for Decentralized Heterogeneous Federated Learningcited by 1
  3. 2026TallyTrain: Communication-Efficient Federated Distillationcited by 0
  4. 2026CALM: Class-wise Agreement and Label-gated Disagreement Modulation for Decentralized Federated Learningcited by 0