R+R:Understanding Hyperparameter Effects in DP-SGD
Not on the map. Why?
Abstract (arXiv)
Research on the effects of essential hyperparameters of DP-SGD lacks consensus, verification, and replication. Contradictory and anecdotal statements on their influence make matters worse. While DP-SGD is the standard optimization algorithm for privacy-preserving machine learning, its adoption is still commonly challenged by low performance compared to non-private learning approaches. As proper hyperparameter settings can improve the privacy-utility trade-off, understanding the influence of the hyperparameters promises to simplify their optimization towards better performance, and likely foster acceptance of private learning. To shed more light on these influences, we conduct a replication study: We synthesize extant research on hyperparameter influences of DP-SGD into conjectures, conduct a dedicated factorial study to independently identify hyperparameter effects, and assess which conjectures can be replicated across multiple datasets, model architectures, and differential privacy budgets. While we cannot (consistently) replicate conjectures about the main and interaction effects of the batch size and the number of epochs, we were able to replicate the conjectured relationship between the clipping threshold and learning rate. Furthermore, we were able to quantify the significant importance of their combination compared to the other hyperparameters.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 21 of its 43 references that are in the map. First posted on arXiv 4 Nov 2024.
Builds on
21 of 43 references.
References · 21
- 2016Densely Connected Convolutional Networkscited by 6,720
- 2016Deep Learning with Differential Privacycited by 2,728
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
- 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
- 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
- 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
- 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
- 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
- 2016Visualizing the Effects of Predictor Variables in Black Box Supervised Learning Modelscited by 213
- 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 153
- 2020Tempered Sigmoid Activations for Deep Learning with Differential Privacycited by 122
9 more
- 2021Large-Scale Differentially Private BERTcited by 118
- 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
- 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
- 2018A General Approach to Adding Differential Privacy to Iterative Training Procedurescited by 84
- 2020Trust in AutoML: Exploring Information Needs for Establishing Trust in Automated Machine Learning Systemscited by 36
- 2021Explaining Hyperparameter Optimization via Partial Dependence Plotscited by 35
- 2021Not all noise is accounted equally: How differentially private learning benefits from large sampling ratescited by 21
- 2018Three Tools for Practical Differential Privacycited by 20
- 2022SmoothNets: Optimizing CNN architecture design for differentially private deep learningcited by 3
Built on by
Citing papers per year: 2025: 1.
1 paper.