# A Differentially Private Framework for Deep Learning with Convexified Loss Functions

arXiv: https://arxiv.org/abs/2204.01049 · HTML: https://mlsanity.com/paper/2204.01049/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 3 Apr 2022.

## Builds on (most cited first, 14 of 14)

- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds](https://mlsanity.com/paper/1605.02065/) (2016)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [Scalable Private Learning with PATE](https://mlsanity.com/paper/1802.08908/) (2018)
- [Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning](https://mlsanity.com/paper/1812.00910/) (2018)
- [Interpretable and Differentially Private Predictions](https://mlsanity.com/paper/1906.02004/) (2019)
- [Effects of Differential Privacy and Data Skewness on Membership Inference Vulnerability](https://mlsanity.com/paper/1911.09777/) (2019)
- [Data and Model Dependencies of Membership Inference Attack](https://mlsanity.com/paper/2002.06856/) (2020)
- [Preserving Differential Privacy in Convolutional Deep Belief Networks](https://mlsanity.com/paper/1706.08839/) (2017)
- [Universal Convexification via Risk-Aversion](https://mlsanity.com/paper/1406.0554/) (2014)

## Built on by (most cited first, 4 of 4)

- [Directional Privacy for Deep Learning](https://mlsanity.com/paper/2211.04686/) (2022)
- [Differentially Private Topological Data Analysis](https://mlsanity.com/paper/2305.03609/) (2023)
- [Additive Logistic Mechanism for Privacy-Preserving Self-Supervised Learning](https://mlsanity.com/paper/2205.12430/) (2022)
- [Practical, Private Assurance of the Value of Collaboration via Fully Homomorphic Encryption](https://mlsanity.com/paper/2310.02563/) (2023)

## Abstract (arXiv)

Differential privacy (DP) has been applied in deep learning for preserving privacy of the underlying training sets. Existing DP practice falls into three categories - objective perturbation, gradient perturbation and output perturbation. They suffer from three main problems. First, conditions on objective functions limit objective perturbation in general deep learning tasks. Second, gradient perturbation does not achieve a satisfactory privacy-utility trade-off due to over-injected noise in each epoch. Third, high utility of the output perturbation method is not guaranteed because of the loose upper bound on the global sensitivity of the trained model parameters as the noise scale parameter. To address these problems, we analyse a tighter upper bound on the global sensitivity of the model parameters. Under a black-box setting, based on this global sensitivity, to control the overall noise injection, we propose a novel output perturbation framework by injecting DP noise into a randomly sampled neuron (via the exponential mechanism) at the output layer of a baseline non-private neural network trained with a convexified loss function. We empirically compare the privacy-utility trade-off, measured by accuracy loss to baseline non-private models and the privacy leakage against black-box membership inference (MI) attacks, between our framework and the open-source differentially private stochastic gradient descent (DP-SGD) approaches on six commonly used real-world datasets. The experimental evaluations show that, when the baseline models have observable privacy leakage under MI attacks, our framework achieves a better privacy-utility trade-off than existing DP-SGD implementations, given an overall privacy budget ε ≤ 1 for a large number of queries.