mlsanityData to 14 Oct 2026

A Masked Pruning Approach for Dimensionality Reduction in Communication-Efficient Federated Learning Systems

arXiv 2312.03889 ↗First posted 6 Dec 2023cs.LG cs.AI cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) represents a growing machine learning (ML) paradigm designed for training models across numerous nodes that retain local datasets, all without directly exchanging the underlying private data with the parameter server (PS). Its increasing popularity is attributed to notable advantages in terms of training deep neural network (DNN) models under privacy aspects and efficient utilization of communication resources. Unfortunately, DNNs suffer from high computational and communication costs, as well as memory consumption in intricate tasks. These factors restrict the applicability of FL algorithms in communication-constrained systems with limited hardware resources. In this paper, we develop a novel algorithm that overcomes these limitations by synergistically combining a pruning-based method with the FL process, resulting in low-dimensional representations of the model with minimal communication cost, dubbed Masked Pruning over FL (MPFL). The algorithm operates by initially distributing weights to the nodes through the PS. Subsequently, each node locally trains its model and computes pruning masks. These low-dimensional masks are then transmitted back to the PS, which generates a consensus pruning mask, broadcasted back to the nodes. This iterative process enhances the robustness and stability of the masked pruning model. The generated mask is used to train the FL model, achieving significant bandwidth savings. We present an extensive experimental study demonstrating the superior performance of MPFL compared to existing methods. Additionally, we have developed an open-source software package for the benefit of researchers and developers in related fields.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 47 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

39 of 47 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2019Advances and Open Problems in Federated Learningcited by 3,046
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  9. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  10. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  11. 2020What is the State of Neural Network Pruning?cited by 583
  12. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 569
27 more
  1. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 378
  2. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  3. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 315
  4. 2019A Joint Learning and Communications Framework for Federated Learning over Wireless Networkscited by 263
  5. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 240
  6. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
  7. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 207
  8. 2015Federated Optimization:Distributed Optimization Beyond the Datacentercited by 195
  9. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 189
  10. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  11. 2017Deep Learning with Low Precision by Half-wave Gaussian Quantizationcited by 117
  12. 2020UVeQFed: Universal Vector Quantization for Federated Learningcited by 107
  13. 2019One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizerscited by 102
  14. 2019Winning the Lottery with Continuous Sparsificationcited by 100
  15. 2020Over-the-Air Federated Learning from Heterogeneous Datacited by 73
  16. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  17. 2019On Analog Gradient Descent Learning over Multiple Access Fading Channelscited by 62
  18. 2021Federated Learning: A Signal Processing Perspectivecited by 49
  19. 2020PoPS: Policy Pruning and Shrinking for Deep Reinforcement Learningcited by 30
  20. 2018Learning to Prune Filters in Convolutional Neural Networkscited by 25
  21. 2021Accelerated Gradient Descent Learning over Multiple Access Fading Channelscited by 14
  22. 2022Revisiting Sparse Convolutional Model for Visual Recognitioncited by 14
  23. 2021Distributed Learning over Markovian Fading Channels for Stable Spectrum Accesscited by 13
  24. 2019VACL: Variance-Aware Cross-Layer Regularization for Pruning Deep Residual Networkscited by 9
  25. 2021Learning in Restless Bandits under Exogenous Global Markov Processcited by 9
  26. 2023Client Selection for Generalization in Accelerated Federated Learning: A Multi-Armed Bandit Approachcited by 8
  27. 2023A Communication-Efficient Adaptive Algorithm for Federated Learning under Cumulative Regretcited by 7

Built on by

Not cited yet.