SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learning
Not on the map. Why?
Abstract (arXiv)
In this work, we propose Salient Sparse Federated Learning (SSFL), a streamlined approach for sparse federated learning with efficient communication. SSFL identifies a sparse subnetwork prior to training, leveraging parameter saliency scores computed separately on local client data in non-IID scenarios, and then aggregated, to determine a global mask. Only the sparse model weights are trained and communicated each round between the clients and the server. On standard benchmarks including CIFAR-10, CIFAR-100, and Tiny-ImageNet, SSFL consistently improves the accuracy sparsity trade off, achieving more than 20% relative error reduction on CIFAR-10 compared to the strongest sparse baseline, while reducing communication costs by 2 × relative to dense FL. Finally, in a real-world federated learning deployment, SSFL delivers over 2.3 × faster communication time, underscoring its practical efficiency.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 61 of its 97 references that are in the map. First posted on arXiv 15 May 2024.
Builds on
61 of 97 references.
References · 61
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2024Mixtral of Expertscited by 1,785
- 2018Group Normalizationcited by 1,514
- 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
- 2016Pruning Filters for Efficient ConvNetscited by 1,129
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
- 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
38 more
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
- 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
- 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
- 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
- 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
- 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
- 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
- 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
- 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
- 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
- 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
- 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
- 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
- 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
- 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
- 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 253
- 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
- 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
- 2021Accelerating Sparse Deep Neural Networkscited by 213
- 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
- 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
- 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
- 2020Sparse GPU Kernels for Deep Learningcited by 138
- 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
- 2019One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizerscited by 102
- 2020The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Modelscited by 91
- 2019Fast Sparse ConvNetscited by 86
- 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
- 2020Progressive Skeletonization: Trimming more fat from a network at initializationcited by 64
- 2023FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Modelscited by 62
- 2020LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasetscited by 60
- 2021Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharingcited by 58
- 2019DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measurescited by 56
- 2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 54
- 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
- 2023ZipLM: Inference-Aware Structured Pruning of Language Modelscited by 46
- 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
- 2020rTop-k: A Statistical Estimation Approach to Distributed SGDcited by 27
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2024: 1.
1 paper.