# Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity

arXiv: https://arxiv.org/abs/2105.00562 · HTML: https://mlsanity.com/paper/2105.00562/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 2 May 2021.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training](https://mlsanity.com/paper/1712.01887/) (2017)
- [Think Locally, Act Globally: Federated Learning with Local and Global Representations](https://mlsanity.com/paper/2001.01523/) (2020)
- [Three Approaches for Personalization with Applications to Federated Learning](https://mlsanity.com/paper/2002.10619/) (2020)
- [Improving Federated Learning Personalization via Model Agnostic Meta Learning](https://mlsanity.com/paper/1909.12488/) (2019)
- [Federated Learning of a Mixture of Global and Local Models](https://mlsanity.com/paper/2002.05516/) (2020)
- [Federated Meta-Learning with Fast Convergence and Efficient Communication](https://mlsanity.com/paper/1802.07876/) (2018)
- [Federated Evaluation of On-device Personalization](https://mlsanity.com/paper/1910.10252/) (2019)
- [Salvaging Federated Learning by Local Adaptation](https://mlsanity.com/paper/2002.04758/) (2020)
- [Geometric Lower Bounds for Distributed Parameter Estimation under Communication Constraints](https://mlsanity.com/paper/1802.08417/) (2018)
- [An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models](https://mlsanity.com/paper/1909.06678/) (2019)
- [Robust and Communication-Efficient Collaborative Learning](https://mlsanity.com/paper/1907.10595/) (2019)

## Built on by (most cited first, 27 of 27)

- [Towards Building the Federated GPT: Federated Instruction Tuning](https://mlsanity.com/paper/2305.05644/) (2023)
- [DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training](https://mlsanity.com/paper/2206.00187/) (2022)
- [Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspaces](https://mlsanity.com/paper/2209.10526/) (2022)
- [Achieving Personalized Federated Learning with Sparse Local Models](https://mlsanity.com/paper/2201.11380/) (2022)
- [Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks](https://mlsanity.com/paper/2209.15595/) (2022)
- [FLIS: Clustered Federated Learning via Inference Similarity for Non-IID Data Distribution](https://mlsanity.com/paper/2208.09754/) (2022)
- [Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents](https://mlsanity.com/paper/2312.01537/) (2023)
- [AEDFL: Efficient Asynchronous Decentralized Federated Learning with Heterogeneous Devices](https://mlsanity.com/paper/2312.10935/) (2023)
- [Complement Sparsification: Low-Overhead Model Pruning for Federated Learning](https://mlsanity.com/paper/2303.06237/) (2023)
- [When Do Curricula Work in Federated Learning?](https://mlsanity.com/paper/2212.12712/) (2022)
- [FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated Clients](https://mlsanity.com/paper/2502.20639/) (2025)
- [FedDebug: Systematic Debugging for Federated Learning Applications](https://mlsanity.com/paper/2301.03553/) (2023)
- [A Practical Recipe for Federated Learning Under Statistical Heterogeneity Experimental Design](https://mlsanity.com/paper/2307.15245/) (2023)
- [Federated Learning Client Pruning for Noisy Labels](https://mlsanity.com/paper/2411.07391/) (2024)
- [CEFHRI: A Communication Efficient Federated Learning Framework for Recognizing Industrial Human-Robot Interaction](https://mlsanity.com/paper/2308.14965/) (2023)
- [Self-Programming Artificial Intelligence Using Code-Generating Language Models](https://mlsanity.com/paper/2205.00167/) (2022)
- [Cross-Silo Federated Learning Across Divergent Domains with Iterative Parameter Alignment](https://mlsanity.com/paper/2311.04818/) (2023)
- [Rapid Deployment of DNNs for Edge Computing via Structured Pruning at Initialization](https://mlsanity.com/paper/2404.16877/) (2024)
- [SSFL: Discovering Sparse Unified Subnetworks at Initialization for Efficient Federated Learning](https://mlsanity.com/paper/2405.09037/) (2024)
- [Federated Learning with Flexible Architectures](https://mlsanity.com/paper/2406.09877/) (2024)
- [Automatic Structured Pruning for Efficient Architecture in Federated Learning](https://mlsanity.com/paper/2411.01759/) (2024)
- [FedBKD: Distilled Federated Learning to Embrace Gerneralization and Personalization on Non-IID Data](https://mlsanity.com/paper/2506.20245/) (2025)
- [Federated Learning based on Pruning and Recovery](https://mlsanity.com/paper/2403.15439/) (2024)
- [subMFL: Compatiple subModel Generation for Federated Learning in Device Heterogenous Environment](https://mlsanity.com/paper/2405.20014/) (2024)
- [Evaluating Small Language Models for Agentic On-Farm Decision Support Systems](https://mlsanity.com/paper/2512.14043/) (2025)
- [FedImp: Enhancing Federated Learning Convergence with Impurity-Based Weighting](https://mlsanity.com/paper/2608.14654/) (2026)
- [FedASAP: Activation Statistics-driven Structured Adaptive Pruning for Efficient Personalized Federated Learning for Lesion Segmentation on brain MRI](https://mlsanity.com/paper/2609.19042/) (2026)

## Abstract (arXiv)

The traditional approach in FL tries to learn a single global model collaboratively with the help of many clients under the orchestration of a central server. However, learning a single global model might not work well for all clients participating in the FL under data heterogeneity. Therefore, the personalization of the global model becomes crucial in handling the challenges that arise with statistical heterogeneity and the non-IID distribution of data. Unlike prior works, in this work we propose a new approach for obtaining a personalized model from a client-level objective. This further motivates all clients to participate in federation even under statistical heterogeneity in order to improve their performance, instead of merely being a source of data and model training for the central server. To realize this personalization, we leverage finding a small subnetwork for each client by applying hybrid pruning (combination of structured and unstructured pruning), and unstructured pruning. Through a range of experiments on different benchmarks, we observed that the clients with similar data (labels) share similar personal parameters. By finding a subnetwork for each client ...