# Improving Deep Learning with Differential Privacy using Gradient Encoding and Denoising

arXiv: https://arxiv.org/abs/2007.11524 · HTML: https://mlsanity.com/paper/2007.11524/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jul 2020.

## Built on by (most cited first, 30 of 30)

- [Deep Learning with Label Differential Privacy](https://mlsanity.com/paper/2102.06062/) (2021)
- [Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture](https://mlsanity.com/paper/2110.08324/) (2021)
- [Split-and-Denoise: Protect large language model inference with local differential privacy](https://mlsanity.com/paper/2310.09130/) (2023)
- [Not all noise is accounted equally: How differentially private learning benefits from large sampling rates](https://mlsanity.com/paper/2110.06255/) (2021)
- [Gradient Leakage Attack Resilient Deep Learning](https://mlsanity.com/paper/2112.13178/) (2021)
- [DPIS: An Enhanced Mechanism for Differentially Private SGD with Importance Sampling](https://mlsanity.com/paper/2210.09634/) (2022)
- [DP-UTIL: Comprehensive Utility Analysis of Differential Privacy in Machine Learning](https://mlsanity.com/paper/2112.12998/) (2021)
- [Differentially Private Federated Learning for Cancer Prediction](https://mlsanity.com/paper/2101.02997/) (2021)
- [MIAShield: Defending Membership Inference Attacks via Preemptive Exclusion of Members](https://mlsanity.com/paper/2203.00915/) (2022)
- [Spectral-DP: Differentially Private Deep Learning through Spectral Perturbation and Filtering](https://mlsanity.com/paper/2307.13231/) (2023)
- [MUDGUARD: Taming Malicious Majorities in Federated Learning using Privacy-Preserving Byzantine-Robust Clustering](https://mlsanity.com/paper/2208.10161/) (2022)
- [Backpropagation Clipping for Deep Learning with Differential Privacy](https://mlsanity.com/paper/2202.05089/) (2022)
- [Learning Privacy-Preserving Student Networks via Discriminative-Generative Distillation](https://mlsanity.com/paper/2409.02404/) (2024)
- [On the Importance of Architecture and Feature Selection in Differentially Private Machine Learning](https://mlsanity.com/paper/2205.06720/) (2022)
- [RQP-SGD: Differential Private Machine Learning through Noisy SGD and Randomized Quantization](https://mlsanity.com/paper/2402.06606/) (2024)
- [Visual Privacy Auditing with Diffusion Models](https://mlsanity.com/paper/2403.07588/) (2024)
- [Privacy-preserving Deep Learning based Record Linkage](https://mlsanity.com/paper/2211.02161/) (2022)
- [Data value estimation on private gradients](https://mlsanity.com/paper/2412.17008/) (2024)
- [AdvSGM: Differentially Private Graph Learning via Adversarial Skip-gram Model](https://mlsanity.com/paper/2503.21426/) (2025)
- [AdaDPIGU: Differentially Private SGD with Adaptive Clipping and Importance-Based Gradient Updates for Deep Neural Networks](https://mlsanity.com/paper/2507.06525/) (2025)
- [Improving the Algorithm of Deep Learning with Differential Privacy](https://mlsanity.com/paper/2107.05457/) (2021)
- [On the Evaluation of User Privacy in Deep Neural Networks using Timing Side Channel](https://mlsanity.com/paper/2208.01113/) (2022)
- [DPAF: Image Synthesis via Differentially Private Aggregation in Forward Phase](https://mlsanity.com/paper/2304.12185/) (2023)
- [Verification of Neural Networks Local Differential Classification Privacy](https://mlsanity.com/paper/2310.20299/) (2023)
- [Training with Differential Privacy: A Gradient-Preserving Noise Reduction Approach with Provable Security](https://mlsanity.com/paper/2409.11663/) (2024)
- [Structure-Preference Enabled Graph Embedding Generation under Differential Privacy](https://mlsanity.com/paper/2501.03451/) (2025)
- [Real-Time Privacy Risk Measurement with Privacy Tokens for Gradient Leakage](https://mlsanity.com/paper/2502.02913/) (2025)
- [Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)](https://mlsanity.com/paper/2508.15141/) (2025)
- [PrivateXR: Defending Privacy Attacks in Extended Reality Through Explainable AI-Guided Differential Privacy](https://mlsanity.com/paper/2512.16851/) (2025)
- [Enhancing Differentially Private Mechanisms via Empirical Bayes](https://mlsanity.com/paper/2606.21107/) (2026)

## Abstract (arXiv)

Deep learning models leak significant amounts of information about their training datasets. Previous work has investigated training models with differential privacy (DP) guarantees through adding DP noise to the gradients. However, such solutions (specifically, DPSGD), result in large degradations in the accuracy of the trained models. In this paper, we aim at training deep learning models with DP guarantees while preserving model accuracy much better than previous works. Our key technique is to encode gradients to map them to a smaller vector space, therefore enabling us to obtain DP guarantees for different noise distributions. This allows us to investigate and choose noise distributions that best preserve model accuracy for a target privacy budget. We also take advantage of the post-processing property of differential privacy by introducing the idea of denoising, which further improves the utility of the trained models without degrading their DP guarantees. We show that our mechanism outperforms the state-of-the-art DPSGD; for instance, for the same model accuracy of 96.1% on MNIST, our technique results in a privacy bound of ε=3.2 compared to ε=6 of DPSGD, which is a significant improvement.