mlsanityData to 14 Oct 2026

Sparse-ProxSkip: Accelerated Sparse-to-Sparse Training in Federated Learning

arXiv 2405.20623 ↗First posted 31 May 2024cs.LG math.OCcited by 0

Not on the map. Why?

Abstract (arXiv)

In Federated Learning (FL), both client resource constraints and communication costs pose major problems for training large models. In the centralized setting, sparse training addresses resource constraints, while in the distributed setting, local training addresses communication costs. Recent work has shown that local training provably improves communication complexity through acceleration. In this work we show that in FL, naive integration of sparse training and acceleration fails, and we provide theoretical and empirical explanations of this phenomenon. We introduce Sparse-ProxSkip, addressing the issue and implementing the efficient technique of Straight-Through Estimator pruning into sparse training. We demonstrate the performance of Sparse-ProxSkip in extensive experiments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

32 of 47 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  4. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  5. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  6. 2018LEAF: A Benchmark for Federated Settingscited by 792
  7. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  8. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  9. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  10. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  11. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  12. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 163
20 more
  1. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  2. 2021On Large-Cohort Training for Federated Learningcited by 74
  3. 2016Training Sparse Neural Networkscited by 73
  4. 2021FedLab: A Flexible Federated Learning Frameworkcited by 55
  5. 2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 55
  6. 2018On the Ineffectiveness of Variance Reduced Optimization for Deep Learningcited by 48
  7. 2022ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsitycited by 45
  8. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  9. 2022Achieving Personalized Federated Learning with Sparse Local Modelscited by 31
  10. 2022RandProx: Primal-Dual Optimization Algorithms with Randomized Proximal Updatescited by 28
  11. 2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
  12. 2017Extended Comparisons of Best Subset Selection, Forward Stepwise Selection, and the Lassocited by 16
  13. 2023JaxPruner: A concise library for sparsity researchcited by 13
  14. 2021FedPrune: Towards Inclusive Federated Learningcited by 12
  15. 2020Federated Nonconvex Sparse Learningcited by 10
  16. 2022Federated Progressive Sparsification (Purge, Merge, Tune)+cited by 8
  17. 2022Federated Pruning: Improving Neural Network Efficiency with Federated Learningcited by 8
  18. 2024FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Modelscited by 5
  19. 2024Gradient-Congruity Guided Federated Sparse Trainingcited by 2
  20. 2024SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learningcited by 1

Built on by

Not cited yet.