Coupling public and private gradient provably helps optimization
Not on the map. Why?
Abstract (arXiv)
The success of large neural networks is crucially determined by the availability of data. It has been observed that training only on a small amount of public data, or privately on the abundant private data can lead to undesirable degradation of accuracy. In this work, we leverage both private and public data to improve the optimization, by coupling their gradients via a weighted linear combination. We formulate an optimal solution for the optimal weight in the convex setting to indicate that the weighting coefficient should be hyperparameter-dependent. Then, we prove the acceleration in the convergence of non-convex loss and the effects of hyper-parameters such as privacy budget, number of iterations, batch size, and model size on the choice of the weighting coefficient. We support our analysis with empirical experiments across language and vision benchmarks, and provide a guideline for choosing the optimal weight of the gradient coupling.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 2 Oct 2023.
Builds on
34 of 45 references.
References · 34
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2016Deep Learning with Differential Privacycited by 2,682
- 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
- 2020Extracting Training Data from Large Language Modelscited by 2,144
- 2019Deep Leakage from Gradientscited by 1,004
- 2021Scaling Vision with Sparse Mixture of Expertscited by 740
- 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
- 2021Differentially Private Fine-tuning of Language Modelscited by 392
22 more
- 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
- 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
- 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
- 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
- 2018Scalable Private Learning with PATEcited by 266
- 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
- 2019Differentially Private Learning with Adaptive Clippingcited by 218
- 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 140
- 2022Reconstructing Training Data from Trained Neural Networkscited by 129
- 2019Deep Learning with Gaussian Differential Privacycited by 112
- 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
- 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 88
- 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
- 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
- 2021Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacycited by 66
- 2022Mixed Differential Privacy in Computer Visioncited by 53
- 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
- 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
- 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
- 2022Private Adaptive Optimization with Side Informationcited by 38
- 2018Three Tools for Practical Differential Privacycited by 20
- 2021Combining Public and Private Datacited by 9
Built on by
Citing papers per year: 2024: 2; 2025: 1.
3 papers.