mlsanityData to 7 Oct 2026

Anarchic Federated Learning

arXiv 2108.09875 ↗First posted 23 Aug 2021cs.LG cs.DCcited by 42

Not on the map. Why?

Abstract (arXiv)

Present-day federated learning (FL) systems deployed over edge networks consists of a large number of workers with high degrees of heterogeneity in data and/or computing capabilities, which call for flexible worker participation in terms of timing, effort, data heterogeneity, etc. To satisfy the need for flexible worker participation, we consider a new FL paradigm called "Anarchic Federated Learning" (AFL) in this paper. In stark contrast to conventional FL models, each worker in AFL has the freedom to choose i) when to participate in FL, and ii) the number of local steps to perform in each round based on its current situation (e.g., battery level, communication channels, privacy concerns). However, such chaotic worker behaviors in AFL impose many new open questions in algorithm design. In particular, it remains unclear whether one could develop convergent AFL training algorithms, and if yes, under what conditions and how fast the achievable convergence speed is. Toward this end, we propose two Anarchic Federated Averaging (AFA) algorithms with two-sided learning rates for both cross-device and cross-silo settings, which are named AFA-CD and AFA-CS, respectively. Somewhat surprisingly, we show that, under mild anarchic assumptions, both AFL algorithms achieve the best known convergence rate as the state-of-the-art algorithms for conventional FL. Moreover, they retain the highly desirable em linear speedup effect with respect of both the number of workers and local steps in the new AFL paradigm. We validate the proposed algorithms with extensive experiments on real-world datasets.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 37 of its 50 references that are in the map. First posted on arXiv 23 Aug 2021.

Builds on

37 of 50 references.

References · 37

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  6. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  7. 2020Adaptive Federated Optimizationcited by 1,036
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  9. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  10. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  11. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  12. 2019Agnostic Federated Learningcited by 451
25 more
  1. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 408
  2. 2019Fair Resource Allocation in Federated Learningcited by 389
  3. 2021A Field Guide to Federated Optimizationcited by 312
  4. 2019Asynchronous Federated Optimizationcited by 264
  5. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  6. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  7. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 178
  8. 2013Minimizing Finite Sums with the Stochastic Average Gradientcited by 164
  9. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  10. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  11. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  12. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 143
  13. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 115
  14. 2018Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithmscited by 106
  15. 2018LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learningcited by 95
  16. 2014Deep learning with Elastic Averaging SGDcited by 90
  17. 2019First Analysis of Local GD on Heterogeneous Datacited by 80
  18. 2021On Large-Cohort Training for Federated Learningcited by 73
  19. 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
  20. 2020Towards Flexible Device Participation in Federated Learningcited by 64
  21. 2018On the Ineffectiveness of Variance Reduced Optimization for Deep Learningcited by 48
  22. 2020Distributed Non-Convex Optimization with Sublinear Speedup under Intermittent Client Availabilitycited by 20
  23. 2018Taming Convergence for Asynchronous Stochastic Gradient Descent with Unbounded Delay in Non-Convex Learningcited by 9
  24. 2013GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Trainingcited by 6
  25. 2020LASG: Lazily Aggregated Stochastic Gradients for Communication-Efficient Distributed Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 11; 2023: 15; 2024: 13; 2025: 1; 2026: 2.

42 papers.

Sort:

  1. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 76
  2. 2022A Unified Analysis of Federated Learning with Arbitrary Client Participationcited by 45
  3. 2023GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgettingcited by 27
  4. 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
  5. 2022A General Theory for Federated Optimization with Asynchronous and Heterogeneous Clients Updatescited by 17
  6. 2023Reconstructing Individual Data Points in Federated Learning Hardened with Differential Privacy and Secure Aggregationcited by 16
  7. 2022Resource Constrained Vehicular Edge Federated Learning with Highly Mobile Connected Vehiclescited by 15
  8. 2023Federated Learning with Regularized Client Participationcited by 13
  9. 2022NET-FLEET: Achieving Linear Convergence Speedup for Fully Decentralized Federated Learning with Heterogeneous Datacited by 12
  10. 2022A Survey on Participant Selection for Federated Learning in Mobile Networkscited by 11
  11. 2023Federated Multi-Objective Learningcited by 11
  12. 2022Taming Fat-Tailed ("Heavier-Tailed'' with Potentially Infinite Variance) Noise in Federated Learningcited by 10
30 more
  1. 2023Hierarchical Federated Learning in Wireless Networks: Pruning Tackles Bandwidth Scarcity and System Heterogeneitycited by 10
  2. 2023Over-the-Air Federated Learning and Optimizationcited by 10
  3. 2024FADAS: Towards Federated Adaptive Asynchronous Optimizationcited by 10
  4. 2022Straggler-Resilient Personalized Federated Learningcited by 9
  5. 2022A Snapshot of the Frontiers of Client Selection in Federated Learningcited by 9
  6. 2024Efficient Federated Learning against Heterogeneous and Non-stationary Client Unavailabilitycited by 9
  7. 2023Federated Minimax Optimization with Client Heterogeneitycited by 8
  8. 2023On the Local Cache Update Rules in Streaming Federated Learningcited by 8
  9. 2024Federated Learning under Periodic Client Participation and Heterogeneous Data: A New Communication-Efficient Algorithm and Analysiscited by 8
  10. 2023Distributed Personalized Empirical Risk Minimizationcited by 6
  11. 2023Federated Learning in the Presence of Adversarial Client Unavailabilitycited by 5
  12. 2024FedStale: leveraging stale client updates in federated learningcited by 5
  13. 2022Combating Client Dropout in Federated Learning via Friend Model Substitutioncited by 3
  14. 2024Empowering Federated Learning with Implicit Gossiping: Mitigating Connection Unreliability Amidst Unknown and Arbitrary Dynamicscited by 3
  15. 2024Understanding Server-Assisted Federated Learning in the Presence of Incomplete Client Participationcited by 3
  16. 2023Keep It Simple: Fault Tolerance Evaluation of Federated Learning with Unreliable Clientscited by 2
  17. 2024Asynchronous Federated Stochastic Optimization for Heterogeneous Objectives Under Arbitrary Delayscited by 2
  18. 2024Privacy Preserving Semi-Decentralized Mean Estimation over Intermittently-Connected Networkscited by 2
  19. 2024Privacy Preserved Blood Glucose Level Cross-Prediction: An Asynchronous Decentralized Federated Learning Approachcited by 2
  20. 2023Federated Gradient Matching Pursuitcited by 1
  21. 2023Joint Client Assignment and UAV Route Planning for Indirect-Communication Federated Learningcited by 1
  22. 2023A Lightweight Method for Tackling Unknown Participation Statistics in Federated Averagingcited by 1
  23. 2023Federated Learning with Reduced Information Leakage and Computationcited by 1
  24. 2024Online-Score-Aided Federated Learning for Resource-Constrained Wireless Clients with Continual Data Arrivalcited by 1
  25. 2022SAGDA: Achieving O(ε-2) Communication Complexity in Federated Min-Max Learningcited by 0
  26. 2024Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networkscited by 0
  27. 2024Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning?cited by 0
  28. 2025Stragglers Can Contribute More: Uncertainty-Aware Distillation for Asynchronous Federated Learningcited by 0
  29. 2026Optimizing Server Placement for Vertical Federated Learning in Dynamic Edge/Fog Networkscited by 0
  30. 2026FedeRage: Provably Convergent Agnostic Federated Learning under General Client Driftcited by 0