mlsanityData to 7 Oct 2026

Communication-Efficient Learning of Deep Networks from Decentralized Data

arXiv 1602.05629 ↗First posted 17 Feb 2016cs.LGcited by 7,592

Not on the map. Why?

Cited by 167 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning. We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.

Key facts. Cited by 7,592 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Feb 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 3; 2021: 864; 2022: 1112; 2023: 1367; 2024: 1578; 2025: 1518; 2026: 1150.

7,592 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  2. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  3. 2021Model-Contrastive Federated Learningcited by 762
  4. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  5. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  6. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  7. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  8. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  9. 2021Towards Personalized Federated Learningcited by 462
  10. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  11. 2021Towards Out-Of-Distribution Generalization: A Surveycited by 431
  12. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
38 more
  1. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  2. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  3. 2021Federated Learning on Non-IID Data: A Surveycited by 345
  4. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 320
  5. 2021A Field Guide to Federated Optimizationcited by 312
  6. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
  7. 2022A Survey of Machine Unlearningcited by 298
  8. 2021Graph Self-Supervised Learning: A Surveycited by 289
  9. 2021FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Spacecited by 284
  10. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  11. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  12. 2022Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representationcited by 243
  13. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  14. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  15. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  16. 2021Personalized Federated Learning using Hypernetworkscited by 210
  17. 2021Trustworthy AI: From Principles to Practicescited by 206
  18. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  19. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  20. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  21. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  22. 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 189
  23. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  24. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  25. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  26. 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
  27. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
  28. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  29. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  30. 2021Practical and Private (Deep) Learning without Sampling or Shufflingcited by 169
  31. 2022Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communicationscited by 166
  32. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
  33. 2024Improving LoRA in Privacy-preserving Federated Learningcited by 158
  34. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  35. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  36. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  37. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  38. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 144

50 most cited of 7,592 All 7,592 (JSON)