mlsanityData to 7 Oct 2026

Demystifying the Effects of Non-Independence in Federated Learning

arXiv 2103.11226 ↗First posted 20 Mar 2021cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) enables statistical models to be built on user-generated data without compromising data security and user privacy. For this reason, FL is well suited for on-device learning from mobile devices where data is abundant and highly privatized. Constrained by the temporal availability of mobile devices, only a subset of devices is accessible to participate in the iterative protocol consisting of training and aggregation. In this study, we take a step toward better understanding the effect of non-independent data distributions arising from block-cyclic sampling. By conducting extensive experiments on visual classification, we measure the effects of block-cyclic sampling (both standalone and in combination with non-balanced block distributions). Specifically, we measure the alterations induced by block-cyclic sampling from the perspective of accuracy, fairness, and convergence rate. Experimental results indicate robustness to cycling over a two-block structure, e.g., due to time zones. In contrast, drawing data samples dependently from a multi-block structure significantly degrades the performance and rate of convergence by up to 26%. Moreover, we find that this performance degeneration is further aggravated by unbalanced block distributions to a point that can no longer be adequately compensated by higher communication and more frequent synchronization.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 20 Mar 2021.

Builds on

35 of 45 references.

References · 35

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2019Advances and Open Problems in Federated Learningcited by 3,021
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  6. 2018Federated Learning with Non-IID Datacited by 1,099
  7. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  8. 2017Focal Loss for Dense Object Detectioncited by 749
  9. 2017Federated Multi-Task Learningcited by 669
  10. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  11. 2019Federated Learning with Personalization Layerscited by 533
  12. 2019Agnostic Federated Learningcited by 451
23 more
  1. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 415
  2. 2019Fair Resource Allocation in Federated Learningcited by 389
  3. 2016Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatmentcited by 368
  4. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  5. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  6. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  7. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  8. 2020Federated Learning of a Mixture of Global and Local Modelscited by 208
  9. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
  10. 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 154
  11. 2019Federated Evaluation of On-device Personalizationcited by 147
  12. 2020Survey of Personalization Techniques for Federated Learningcited by 144
  13. 2019Overcoming Forgetting in Federated Learning on Non-IID Datacited by 120
  14. 2019Federated Learning for Emoji Prediction in a Mobile Keyboardcited by 115
  15. 2019Variational Federated Multi-Task Learningcited by 69
  16. 2020Federated learning with class imbalance reductioncited by 52
  17. 2019Semi-Cyclic Stochastic Gradient Descentcited by 40
  18. 2020Fairness and Accuracy in Federated Learningcited by 27
  19. 2020Fed-Focal Loss for imbalanced data classification in Federated Learningcited by 26
  20. 2020Specialized federated learning using a mixture of expertscited by 23
  21. 2020FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learningcited by 14
  22. 2020Addressing Class Imbalance in Federated Learningcited by 6
  23. 2018Stochastic Distributed Optimization for Machine Learning from Decentralized Featurescited by 2

Built on by

Cited by year

Citing papers per year: 2021: 1.

1 paper.

  1. 2021Robust Convergence in Federated Learning through Label-wise Clusteringcited by 2