mlsanityData to 7 Oct 2026

A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization

arXiv 2212.04486 ↗First posted 8 Dec 2022cs.LG cs.AI cs.CRcited by 11

Not on the map. Why?

Abstract (arXiv)

An open problem in differentially private deep learning is hyperparameter optimization (HPO). DP-SGD introduces new hyperparameters and complicates existing ones, forcing researchers to painstakingly tune hyperparameters with hundreds of trials, which in turn makes it impossible to account for the privacy cost of HPO without destroying the utility. We propose an adaptive HPO method that uses cheap trials (in terms of privacy cost and runtime) to estimate optimal hyperparameters and scales them up. We obtain state-of-the-art performance on 22 benchmark tasks, across computer vision and natural language processing, across pretraining and finetuning, across architectures and a wide range of varepsilon ∈ [0.01,8.0], all while accounting for the privacy cost of HPO.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 49 of its 82 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

49 of 82 references.

References · 49

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  3. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2016Deep Learning with Differential Privacycited by 2,682
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  12. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
37 more
  1. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  2. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  3. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  4. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  5. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  6. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  7. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  8. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  9. 2017Functional Map of the Worldcited by 338
  10. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 234
  11. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  12. 2017EMNIST: an extension of MNIST to handwritten letterscited by 212
  13. 2021Numerical Composition of Differential Privacycited by 174
  14. 2018Amplification by Shuffling: From Local to Central Differential Privacy via Anonymitycited by 163
  15. 2020Momentum Improves Normalized SGDcited by 136
  16. 2022Differentially Private Diffusion Modelscited by 118
  17. 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
  18. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  19. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  20. 2018Private Selection from Private Candidatescited by 75
  21. 2019Distributionally Robust Optimization: A Reviewcited by 75
  22. 2020Rethinking the Hyperparameters for Fine-tuningcited by 69
  23. 2022Recovering Private Text in Federated Learning of Language Modelscited by 67
  24. 2021A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustnesscited by 62
  25. 2022TAN Without a Burn: Scaling Laws of DP-SGDcited by 51
  26. 2021Public Data-Assisted Mirror Descent for Private Model Trainingcited by 49
  27. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  28. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
  29. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 47
  30. 2021SparseFed: Mitigating Model Poisoning Attacks in Federated Learning with Sparsificationcited by 46
  31. 2019Augment your batch: better training with larger batchescited by 37
  32. 2022Just Fine-tune Twice: Selective Differential Privacy for Large Language Modelscited by 32
  33. 2021The Role of Adaptive Optimizers for Honest Private Hyperparameter Selectioncited by 25
  34. 2022How to Fine-Tune Vision Models with SGDcited by 23
  35. 2022Differentially Private Adaptive Optimization with Delayed Preconditionerscited by 18
  36. 2022Fine-Tuning with Differential Privacy Necessitates an Additional Hyperparameter Searchcited by 15
  37. 2022Differentially Private Image Classification from Featurescited by 7

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 2; 2025: 5.

11 papers.

  1. 2023Privacy-Preserving In-Context Learning for Large Language Modelscited by 50
  2. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  3. 2023Selective Pre-training for Private Fine-tuningcited by 22
  4. 2025Privacy Auditing of Large Language Modelscited by 22
  5. 2023Differentially Private Sharpness-Aware Trainingcited by 14
  6. 2023DP-Mix: Mixup-based Data Augmentation for Differentially Private Learningcited by 9
  7. 2025Towards hyperparameter-free optimization with differential privacycited by 7
  8. 2024Privacy Amplification for the Gaussian Mechanism via Bounded Supportcited by 2
  9. 2025Spurious Privacy Leakage in Neural Networkscited by 1
  10. 2025An Interactive Framework for Finding the Optimal Trade-off in Differential Privacycited by 1
  11. 2025On Optimal Hyperparameters for Differentially Private Deep Transfer Learningcited by 1