mlsanityData to 7 Oct 2026

On the Convergence of Heterogeneous Federated Learning with Arbitrary Adaptive Online Model Pruning

arXiv 2201.11803 ↗First posted 27 Jan 2022cs.LG cs.DCcited by 4

Not on the map. Why?

Abstract (arXiv)

One of the biggest challenges in Federated Learning (FL) is that client devices often have drastically different computation and communication resources for local updates. To this end, recent research efforts have focused on training heterogeneous local models obtained by pruning a shared global model. Despite empirical success, theoretical guarantees on convergence remain an open question. In this paper, we present a unifying framework for heterogeneous FL algorithms with em arbitrary adaptive online model pruning and provide a general convergence analysis. In particular, we prove that under certain sufficient conditions and on both IID and non-IID data, these algorithms converges to a stationary point of standard FL for general smooth cost functions, with a convergence rate of O(frac1√Q). Moreover, we illuminate two key factors impacting convergence: pruning-induced noise and minimum coverage index, advocating a joint design of local pruning masks for efficient training.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 27 Jan 2022.

Builds on

37 of 49 references.

References · 37

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  3. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  4. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  5. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  7. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  8. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  9. 2019Towards Federated Learning at Scale: System Designcited by 826
  10. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  11. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  12. 2018Rethinking the Value of Network Pruningcited by 595
25 more
  1. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  2. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  3. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 408
  4. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 407
  5. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  6. 2020SplitFed: When Federated Learning Meets Split Learningcited by 308
  7. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  8. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  9. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  10. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
  11. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  12. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 162
  13. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  14. 2020Dynamic Model Pruning with Feedbackcited by 125
  15. 2018Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithmscited by 106
  16. 2019One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizerscited by 102
  17. 2019Communication-efficient distributed SGD with Sketchingcited by 81
  18. 2020Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approachcited by 80
  19. 2018Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGDcited by 76
  20. 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
  21. 2018Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimizationcited by 34
  22. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  23. 2021Effective Model Sparsification by Scheduled Grow-and-Prune Methodscited by 25
  24. 2019Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaborationcited by 15
  25. 2019Privacy Preserving Stochastic Channel-Based Federated Learning with Neural Network Pruningcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 1.

4 papers.

  1. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  2. 2023Tackling Computational Heterogeneity in FL: A Few Theoretical Insightscited by 0
  3. 2023Adaptive pruning-based Newton's method for distributed learningcited by 0
  4. 2024subMFL: Compatiple subModel Generation for Federated Learning in Device Heterogenous Environmentcited by 0