# Differentially Private Federated Learning with Local Regularization and Sparsification

arXiv: https://arxiv.org/abs/2203.03106 · HTML: https://mlsanity.com/paper/2203.03106/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 7 Mar 2022.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training](https://mlsanity.com/paper/1712.01887/) (2017)
- [Differentially Private Federated Learning: A Client Level Perspective](https://mlsanity.com/paper/1712.07557/) (2017)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [Differentially Private Learning with Adaptive Clipping](https://mlsanity.com/paper/1905.03871/) (2019)
- [cpSGD: Communication-efficient and differentially-private distributed SGD](https://mlsanity.com/paper/1805.10559/) (2018)
- [The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation](https://mlsanity.com/paper/2102.06387/) (2021)
- [Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacy](https://mlsanity.com/paper/2106.13673/) (2021)
- [LDP-FL: Practical Private Aggregation in Federated Learning with Local Differential Privacy](https://mlsanity.com/paper/2007.15789/) (2020)
- [Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communication](https://mlsanity.com/paper/1805.08768/) (2018)
- [Federated Model Distillation with Noise-Free Differential Privacy](https://mlsanity.com/paper/2009.05537/) (2020)
- [Federated Learning with Sparsification-Amplified Privacy and Adaptive Optimization](https://mlsanity.com/paper/2008.01558/) (2020)
- [Variance-based Gradient Compression for Efficient Distributed Deep Learning](https://mlsanity.com/paper/1802.06058/) (2018)
- [Voting-based Approaches For Differentially Private Federated Learning](https://mlsanity.com/paper/2010.04851/) (2020)

## Built on by (most cited first, 30 of 36)

- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [No One Left Behind: Real-World Federated Class-Incremental Learning](https://mlsanity.com/paper/2302.00903/) (2023)
- [Eliminating Domain Bias for Federated Learning in Representation Space](https://mlsanity.com/paper/2311.14975/) (2023)
- [Make Landscape Flatter in Differentially Private Federated Learning](https://mlsanity.com/paper/2303.11242/) (2023)
- [DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2309.13546/) (2023)
- [Threats and Defenses in Federated Learning Life Cycle: A Comprehensive Survey and Challenges](https://mlsanity.com/paper/2407.06754/) (2024)
- [Amplitude-Varying Perturbation for Balancing Privacy and Utility in Federated Learning](https://mlsanity.com/paper/2303.04274/) (2023)
- [Prototype Correlation Matching and Class-Relation Reasoning for Few-Shot Medical Image Segmentation](https://mlsanity.com/paper/2406.05054/) (2024)
- [Unraveling the Connections between Privacy and Certified Robustness in Federated Learning Against Poisoning Attacks](https://mlsanity.com/paper/2209.04030/) (2022)
- [FedUV: Uniformity and Variance for Heterogeneous Federated Learning](https://mlsanity.com/paper/2402.18372/) (2024)
- [DP-FedPGN: Finding Global Flat Minima for Differentially Private Federated Learning via Penalizing Gradient Norm](https://mlsanity.com/paper/2510.27504/) (2025)
- [Clients Collaborate: Flexible Differentially Private Federated Learning with Guaranteed Improvement of Utility-Privacy Trade-off](https://mlsanity.com/paper/2402.07002/) (2024)
- [Layered Randomized Quantization for Communication-Efficient and Privacy-Preserving Distributed Learning](https://mlsanity.com/paper/2312.07060/) (2023)
- [Communication and Energy Efficient Wireless Federated Learning with Intrinsic Privacy](https://mlsanity.com/paper/2304.07460/) (2023)
- [Towards the Flatter Landscape and Better Generalization in Federated Learning under Client-level Differential Privacy](https://mlsanity.com/paper/2305.00873/) (2023)
- [Efficient Federated Learning with Enhanced Privacy via Lottery Ticket Pruning in Edge Computing](https://mlsanity.com/paper/2305.01387/) (2023)
- [Dynamic Privacy Allocation for Locally Differentially Private Federated Learning with Composite Objectives](https://mlsanity.com/paper/2308.01139/) (2023)
- [Clip Body and Tail Separately: High Probability Guarantees for DPSGD with Heavy Tails](https://mlsanity.com/paper/2405.17529/) (2024)
- [FedCFA: Alleviating Simpson's Paradox in Model Aggregation with Counterfactual Federated Learning](https://mlsanity.com/paper/2412.18904/) (2024)
- [Privacy Preservation in Artificial Intelligence and Extended Reality (AI-XR) Metaverses: A Survey](https://mlsanity.com/paper/2310.10665/) (2023)
- [Federated Learning for Smart Grid: A Survey on Applications and Potential Vulnerabilities](https://mlsanity.com/paper/2409.10764/) (2024)
- [Review of Mathematical Optimization in Federated Learning](https://mlsanity.com/paper/2412.01630/) (2024)
- [PCDP-SGD: Improving the Convergence of Differentially Private SGD via Projection in Advance](https://mlsanity.com/paper/2312.03792/) (2023)
- [Privacy-Preserving Federated Learning with Consistency via Knowledge Distillation Using Conditional Generator](https://mlsanity.com/paper/2409.06955/) (2024)
- [Optimal Client Sampling in Federated Learning with Client-Level Heterogeneous Differential Privacy](https://mlsanity.com/paper/2505.13655/) (2025)
- [Cyst-X: A Multi-Center MRI Benchmark and Federated Learning Framework for Malignancy-Risk Stratification of Pancreatic Cystic Neoplasm](https://mlsanity.com/paper/2507.22017/) (2025)
- [DA-PFL: Dynamic Affinity Aggregation for Personalized Federated Learning](https://mlsanity.com/paper/2403.09284/) (2024)
- [Federated Hypergraph Learning with Local Differential Privacy: Toward Privacy-Aware Hypergraph Structure Completion](https://mlsanity.com/paper/2408.05160/) (2024)
- [Convergent Differential Privacy Analysis for General Federated Learning](https://mlsanity.com/paper/2408.15621/) (2024)
- [DP2-FedSAM: Enhancing Differentially Private Federated Learning Through Personalized Sharpness-Aware Minimization](https://mlsanity.com/paper/2409.13645/) (2024)

## Abstract (arXiv)

User-level differential privacy (DP) provides certifiable privacy guarantees to the information that is specific to any user's data in federated learning. Existing methods that ensure user-level DP come at the cost of severe accuracy decrease. In this paper, we study the cause of model performance degradation in federated learning under user-level DP guarantee. We find the key to solving this issue is to naturally restrict the norm of local updates before executing operations that guarantee DP. To this end, we propose two techniques, Bounded Local Update Regularization and Local Update Sparsification, to increase model quality without sacrificing privacy. We provide theoretical analysis on the convergence of our framework and give rigorous privacy guarantees. Extensive experiments show that our framework significantly improves the privacy-utility trade-off over the state-of-the-arts for federated learning with user-level DP guarantee.