mlsanityData to 14 Oct 2026

Towards hyperparameter-free optimization with differential privacy

arXiv 2503.00703 ↗First posted 2 Mar 2025cs.LG cs.CL cs.CRcited by 7

Not on the map. Why?

Abstract (arXiv)

Differential privacy (DP) is a privacy-preserving paradigm that protects the training data when training deep learning models. Critically, the performance of models is determined by the training hyperparameters, especially those of the learning rate schedule, thus requiring fine-grained hyperparameter tuning on the data. In practice, it is common to tune the learning rate hyperparameters through the grid search that (1) is computationally expensive as multiple runs are needed, and (2) increases the risk of data leakage as the selection of hyperparameters is data-dependent. In this work, we adapt the automatic learning rate schedule to DP optimization for any models and optimizers, so as to significantly mitigate or even eliminate the cost of hyperparameter tuning when applied together with automatic per-sample gradient clipping. Our hyperparameter-free DP optimization is almost as computationally efficient as the standard non-DP optimization, and achieves state-of-the-art DP performance on various language and vision tasks.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 32 of its 36 references that are in the map. First posted on arXiv 2 Mar 2025.

Builds on

32 of 36 references.

References · 32

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2016Deep Learning with Differential Privacycited by 2,728
  3. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  4. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  5. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  6. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  7. 2023Fine-Tuning Language Models with Just Forward Passescited by 345
  8. 2024LoRA Learns Less and Forgets Lesscited by 328
  9. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 224
  10. 2019Differentially Private Learning with Adaptive Clippingcited by 220
  11. 2023Prodigy: An Expeditiously Adaptive Parameter-Free Learnercited by 133
  12. 2019Deep Learning with Gaussian Differential Privacycited by 114
20 more
  1. 2023Learning-Rate-Free Learning by D-Adaptationcited by 113
  2. 2021Hyperparameter Tuning with Renyi Differential Privacycited by 104
  3. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 88
  4. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  5. 2023DoG is SGD's Best Friend: A Parameter-Free Dynamic Step Size Schedulecited by 85
  6. 2018Private Selection from Private Candidatescited by 75
  7. 2020CoinPress: Practical Private Mean and Covariance Estimationcited by 74
  8. 2020Private Mean Estimation of Heavy-Tailed Distributionscited by 73
  9. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 48
  10. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  11. 2022Normalized/Clipped SGD with Perturbation for Differentially Private Non-Convex Optimizationcited by 36
  12. 2023DoWG Unleashed: An Efficient Universal Parameter-Free Gradient Descent Methodcited by 35
  13. 2021The Role of Adaptive Optimizers for Honest Private Hyperparameter Selectioncited by 25
  14. 2023Practical Differentially Private Hyperparameter Tuning with Subsamplingcited by 19
  15. 2024Accelerated Parameter-Free Stochastic Optimizationcited by 15
  16. 2024Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuningcited by 14
  17. 2022A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimizationcited by 11
  18. 2024Gradient descent with generalized Newton's methodcited by 9
  19. 2023On the accuracy and efficiency of group-wise clipping in differentially private optimizationcited by 7
  20. 2023DP-HyPO: An Adaptive Private Hyperparameter Optimization Frameworkcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2026On the Price of Privacy for Language Identification and Generationcited by 3
  2. 2025Accelerating Differentially Private Federated Learning via Adaptive Extrapolationcited by 1
  3. 2025DP-DocLDM: Differentially Private Document Image Generation using Latent Diffusion Modelscited by 1
  4. 2025An Interactive Framework for Finding the Optimal Trade-off in Differential Privacycited by 1
  5. 2025On Optimal Hyperparameters for Differentially Private Deep Transfer Learningcited by 1
  6. 2025PrAda-GAN: A Private Adaptive Generative Adversarial Network with Bayes Network Structurecited by 1
  7. 2026DP-LAC: Lightweight Adaptive Clipping for Differentially Private Federated Fine-tuning of Language Modelscited by 0