mlsanityData to 7 Oct 2026

Sparse Random Networks for Communication-Efficient Federated Learning

arXiv 2209.15328 ↗First posted 30 Sep 2022cs.LG stat.AP stat.MLcited by 32

Not on the map. Why?

Abstract (arXiv)

One main challenge in federated learning is the large communication cost of exchanging weight updates from clients to the server at each round. While prior work has made great progress in compressing the weight updates through gradient compression methods, we propose a radically different approach that does not update the weights at all. Instead, our method freezes the weights at their initial random values and learns how to sparsify the random network for the best performance. To this end, the clients collaborate in training a stochastic binary mask to find the optimal sparse random network within the original one. At the end of the training, the final model is a sparse network with random weights -- or a subnetwork inside the dense random network. We show improvements in accuracy, communication (less than 1 bit per parameter (bpp)), convergence speed, and final model size (less than 1 bpp) over relevant baselines on MNIST, EMNIST, CIFAR-10, and CIFAR-100 datasets, in the low bitrate regime under various system configurations.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 54 of its 72 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

54 of 72 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  8. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  9. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  10. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  11. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 415
  12. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 349
38 more
  1. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
  2. 2017Sparse Communication for Distributed Gradient Descentcited by 243
  3. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 237
  4. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  5. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 234
  6. 2019PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimizationcited by 234
  7. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
  8. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  9. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
  10. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 178
  11. 2018Amplification by Shuffling: From Local to Central Differential Privacy via Anonymitycited by 163
  12. 2020Characterising Bias in Compressed Modelscited by 162
  13. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 159
  14. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 135
  15. 2016Distributed Mean Estimation with Limited Communicationcited by 134
  16. 2020Hiding Among the Clones: A Simple and Nearly Optimal Analysis of Privacy Amplification by Shufflingcited by 96
  17. 2020Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficientcited by 81
  18. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  19. 2021The Skellam Mechanism for Differentially Private Federated Learningcited by 66
  20. 2020LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasetscited by 60
  21. 2021Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Networkcited by 55
  22. 2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 54
  23. 2021EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learningcited by 53
  24. 2021DRIVE: One-bit Distributed Mean Estimationcited by 52
  25. 2020Privacy Amplification via Random Check-Inscited by 39
  26. 2016Ternary Neural Networks for Resource-Efficient AI Applicationscited by 30
  27. 2020rTop-k: A Statistical Estimation Approach to Distributed SGDcited by 27
  28. 2021Time-Correlated Sparsification for Communication-Efficient Federated Learningcited by 25
  29. 2021An Information-Theoretic Justification for Model Pruningcited by 23
  30. 2020FedSKETCH: Communication-Efficient and Private Federated Learning via Sketchingcited by 20
  31. 2020Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learningcited by 16
  32. 2021Masked Training of Neural Networks with Partial Gradientscited by 15
  33. 2022Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theorycited by 14
  34. 2021FedPrune: Towards Inclusive Federated Learningcited by 12
  35. 2021Communication-Efficient and Personalized Federated Lottery Ticket Learningcited by 10
  36. 2021FRL: Federated Rank Learningcited by 8
  37. 2022QUIC-FL: Quick Unbiased Compression for Federated Learningcited by 7
  38. 2022HideNseek: Federated Lottery Ticket via Server-side Pruning and Sign Supermaskcited by 6

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 18; 2025: 6; 2026: 1.

32 papers.

Sort:

  1. 2024Advances and Open Challenges in Federated Foundation Modelscited by 33
  2. 2024A Survey on Efficient Federated Learning Methods for Foundation Model Trainingcited by 29
  3. 2024Federated LoRA with Sparse Communicationcited by 26
  4. 2024FIARSE: Model-Heterogeneous Federated Learning via Importance-Aware Submodel Extractioncited by 21
  5. 2023Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomnesscited by 18
  6. 2023Exact Optimality of Communication-Privacy-Utility Tradeoffs in Distributed Mean Estimationcited by 9
  7. 2024SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overheadcited by 9
  8. 2024FedBAT: Communication-Efficient Federated Learning via Learnable Binarizationcited by 9
  9. 2023Adaptive Compression in Federated Learning via Side Informationcited by 6
  10. 2023LASER: Linear Compression in Wireless Distributed Optimizationcited by 6
  11. 2025The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learningcited by 5
  12. 2024Improved Communication-Privacy Trade-offs in L2 Mean Estimation under Streaming Differential Privacycited by 4
20 more
  1. 2024Masked Random Noise for Communication Efficient Federated Learningcited by 4
  2. 2024Decentralized Personalized Federated Learning based on a Conditional Sparse-to-Sparser Schemecited by 3
  3. 2024Unity is Power: Semi-Asynchronous Collaborative Training of Large-Scale Models with Structured Pruning in Resource-Limited Clientscited by 3
  4. 2024On the Sparsity of the Strong Lottery Ticket Hypothesiscited by 3
  5. 2025PRISM: Privacy-Preserving Improved Stochastic Masking for Federated Generative Modelscited by 3
  6. 2023Federated Fine-Tuning of Foundation Models via Probabilistic Maskingcited by 2
  7. 2024FedPeWS: Personalized Warmup via Subnetworks for Enhanced Heterogeneous Federated Learningcited by 2
  8. 2023A Comprehensive View of Personalized Federated Learning on Heterogeneous Clinical Datasetscited by 1
  9. 2024Learn What You Need in Personalized Federated Learningcited by 1
  10. 2024SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learningcited by 1
  11. 2024Automatic Structured Pruning for Efficient Architecture in Federated Learningcited by 1
  12. 2025FedBiF: Communication-Efficient Federated Learning via Bits Freezingcited by 1
  13. 2023Communication-Efficient Federated Learning via Regularized Sparse Random Networkscited by 0
  14. 2024Sparse-ProxSkip: Accelerated Sparse-to-Sparse Training in Federated Learningcited by 0
  15. 2024UniFed: A Universal Federation of a Mixture of Highly Heterogeneous Medical Image Classification Taskscited by 0
  16. 2024Getting Free Bits Back from Rotational Symmetries in LLMscited by 0
  17. 2025Towards Privacy-Preserving and Heterogeneity-aware Split Federated Learning via Probabilistic Maskingcited by 0
  18. 2025ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Modelscited by 0
  19. 2025Learned Digital Codes for Over-the-Air Computation in Federated Edge Learningcited by 0
  20. 2026Representation-Aligned Multi-Scale Personalization for Federated Learningcited by 0