# Complement Sparsification: Low-Overhead Model Pruning for Federated Learning

arXiv: https://arxiv.org/abs/2303.06237 · HTML: https://mlsanity.com/paper/2303.06237/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 27 references that are in the map.
- First posted on arXiv 10 Mar 2023.

## Builds on (most cited first, 19 of 19)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training](https://mlsanity.com/paper/1712.01887/) (2017)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [Adaptive Federated Learning in Resource Constrained Edge Computing Systems](https://mlsanity.com/paper/1804.05271/) (2018)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [Model Pruning Enables Efficient Federated Learning on Edge Devices](https://mlsanity.com/paper/1909.12326/) (2019)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach](https://mlsanity.com/paper/2001.04756/) (2020)
- [Pruning from Scratch](https://mlsanity.com/paper/1909.12579/) (2019)
- [LotteryFL: Personalized and Communication-Efficient Federated Learning with Lottery Ticket Hypothesis on Non-IID Datasets](https://mlsanity.com/paper/2008.03371/) (2020)
- [Federated Dropout -- A Simple Approach for Enabling Federated Learning on Resource Constrained Devices](https://mlsanity.com/paper/2109.15258/) (2021)
- [The Generalization-Stability Tradeoff In Neural Network Pruning](https://mlsanity.com/paper/1906.03728/) (2019)
- [Mitigating Backdoor Attacks in Federated Learning](https://mlsanity.com/paper/2011.01767/) (2020)
- [QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning](https://mlsanity.com/paper/2107.13892/) (2021)
- [Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity](https://mlsanity.com/paper/2105.00562/) (2021)

## Built on by (most cited first, 12 of 12)

- [FedSpaLLM: Federated Pruning of Large Language Models](https://mlsanity.com/paper/2410.14852/) (2024)
- [A Survey on Foundation Models for Personalized Federated Intelligence](https://mlsanity.com/paper/2505.06907/) (2025)
- [Learnable Sparse Customization in Heterogeneous Edge Computing](https://mlsanity.com/paper/2412.07216/) (2024)
- [Energy-Efficient and Real-Time Sensing for Federated Continual Learning via Sample-Driven Control](https://mlsanity.com/paper/2310.07497/) (2023)
- [Concept Matching: Clustering-based Federated Continual Learning](https://mlsanity.com/paper/2311.06921/) (2023)
- [Skewness-Guided Pruning of Multimodal Swin Transformers for Federated Skin Lesion Classification on Edge Devices](https://mlsanity.com/paper/2512.08751/) (2025)
- [CA-HFP: Curvature-Aware Heterogeneous Federated Pruning with Model Reconstruction](https://mlsanity.com/paper/2603.12591/) (2026)
- [FedX: Adaptive Model Decomposition and Quantization for IoT Federated Learning](https://mlsanity.com/paper/2504.12849/) (2025)
- [Addressing the Collaboration Dilemma in Low-Data Federated Learning via Transient Sparsity](https://mlsanity.com/paper/2506.00932/) (2025)
- [Enhancing Communication Efficiency in FL with Adaptive Gradient Quantization and Communication Frequency Optimization](https://mlsanity.com/paper/2509.23419/) (2025)
- [CEPerFed: Communication-Efficient Personalized Federated Learning for Multi-Pulse MRI Classification](https://mlsanity.com/paper/2510.17584/) (2025)
- [Upholding Robustness in Federated Learning: Trends, Emerging Strategies, and Research Opportunities](https://mlsanity.com/paper/2609.28722/) (2026)

## Abstract (arXiv)

Federated Learning (FL) is a privacy-preserving distributed deep learning paradigm that involves substantial communication and computation effort, which is a problem for resource-constrained mobile and IoT devices. Model pruning/sparsification develops sparse models that could solve this problem, but existing sparsification solutions cannot satisfy at the same time the requirements for low bidirectional communication overhead between the server and the clients, low computation overhead at the clients, and good model accuracy, under the FL assumption that the server does not have access to raw data to fine-tune the pruned models. We propose Complement Sparsification (CS), a pruning mechanism that satisfies all these requirements through a complementary and collaborative pruning done at the server and the clients. At each round, CS creates a global sparse model that contains the weights that capture the general data distribution of all clients, while the clients create local sparse models with the weights pruned from the global model to capture the local trends. For improved model performance, these two types of complementary sparse models are aggregated into a dense model in each round, which is subsequently pruned in an iterative process. CS requires little computation overhead on the top of vanilla FL for both the server and the clients. We demonstrate that CS is an approximation of vanilla FL and, thus, its models perform well. We evaluate CS experimentally with two popular FL benchmark datasets. CS achieves substantial reduction in bidirectional communication, while achieving performance comparable with vanilla FL. In addition, CS outperforms baseline pruning mechanisms for FL.