mlsanityData to 7 Oct 2026

Private Fine-tuning of Large Language Models with Zeroth-order Optimization

arXiv 2401.04343 ↗First posted 9 Jan 2024cs.LG cs.CL cs.CRcited by 41

Not on the map. Why?

Abstract (arXiv)

Differentially private stochastic gradient descent (DP-SGD) allows models to be trained in a privacy-preserving manner, but has proven difficult to scale to the era of foundation models. We introduce DP-ZO, a private fine-tuning framework for large language models by privatizing zeroth order optimization methods. A key insight into the design of our method is that the direction of the gradient in the zeroth-order optimization we use is random and the only information from training data is the step size, i.e., a scalar. Therefore, we only need to privatize the scalar step size, which is memory-efficient. DP-ZO provides a strong privacy-utility trade-off across different tasks, and model sizes that are comparable to DP-SGD in (varepsilon,δ)-DP. Notably, DP-ZO possesses significant advantages over DP-SGD in memory efficiency, and obtains higher utility in varepsilon-DP when using the Laplace mechanism.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 60 of its 96 references that are in the map. First posted on arXiv 9 Jan 2024.

Builds on

60 of 96 references.

References · 60

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023Mistral 7Bcited by 3,065
  10. 2016Deep Learning with Differential Privacycited by 2,682
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
38 more
  1. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  2. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  3. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 716
  4. 2017Renyi Differential Privacycited by 643
  5. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  6. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  7. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  8. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  9. 2013The Composition Theorem for Differential Privacycited by 290
  10. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 237
  11. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  12. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  13. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  14. 2018Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoisingcited by 216
  15. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 190
  16. 2021Numerical Composition of Differential Privacycited by 174
  17. 2020Stability of Stochastic Gradient Descent on Nonsmooth Convex Lossescited by 172
  18. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  19. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 160
  20. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  21. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 140
  22. 2016Concentrated Differential Privacycited by 139
  23. 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 109
  24. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 94
  25. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  26. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
  27. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  28. 2021Optimal Accounting of Differential Privacy via Characteristic Functioncited by 83
  29. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 80
  30. 2022Differentially Private Optimization on Large Model at Small Costcited by 65
  31. 2023DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Passcited by 65
  32. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  33. 2022Connect the Dots: Tighter Discrete Approximations of Privacy Loss Distributionscited by 50
  34. 2023Privacy-Preserving In-Context Learning for Large Language Modelscited by 50
  35. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  36. 2022Differentially Private Bias-Term Fine-tuning of Foundation Modelscited by 47
  37. 2020Tight Differential Privacy for Discrete-Valued Mechanisms and for the Subsampled Gaussian Mechanism Using FFTcited by 43
  38. 2018DP-ADMM: ADMM-based Distributed Learning with Differential Privacycited by 41

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 21; 2026: 11.

41 papers.

Sort:

By area:

  1. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 34
  2. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  3. 2025Privacy Auditing of Large Language Modelscited by 22
  4. 2024AI Risk Management Should Incorporate Both Safety and Securitycited by 17
  5. 2024Scalable DP-SGD: Shuffling vs. Poisson Subsamplingcited by 16
  6. 2024Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuningcited by 14
  7. 2024Balls-and-Bins Sampling for DP-SGDcited by 13
  8. 2024Data-adaptive Differentially Private Prompt Synthesis for In-Context Learningcited by 10
  9. 2025Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuningcited by 10
  10. 2025Towards hyperparameter-free optimization with differential privacycited by 7
  11. 2024Privacy-preserving Fine-tuning of Large Language Models through Flatnesscited by 6
  12. 2025Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Trainingcited by 5
29 more
  1. 2024Communication-Efficient Byzantine-Resilient Federated Zero-Order Optimizationcited by 4
  2. 2025ElasticZO: A Memory-Efficient On-Device Learning with Combined Zeroth- and First-Order Optimizationcited by 4
  3. 2025KerZOO: Kernel Function Informed Zeroth-Order Optimization for Accurate and Accelerated LLM Fine-Tuningcited by 4
  4. 2025Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Modelscited by 4
  5. 2025The Hitchhiker's Guide to Efficient, End-to-End, and Tight DP Auditingcited by 4
  6. 2026Fundamental Limitations of Favorable Privacy-Utility Guarantees for DP-SGDcited by 4
  7. 2025Private Text Generation by Seeding Large Language Model Promptscited by 2
  8. 2025Memory-Efficient Differentially Private Training with Gradient Random Projectioncited by 2
  9. 2025Towards Fast LLM Fine-tuning through Zeroth-Order Optimization with Projected Gradient-Aligned Perturbationscited by 2
  10. 2025On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimizationcited by 2
  11. 2025Private Zeroth-Order Optimization with Public Datacited by 2
  12. 2025Understanding Private Learning From Feature Perspectivecited by 2
  13. 2026Q-ShiftDP: A Differentially Private Parameter-Shift Rule for Quantum Machine Learningcited by 1
  14. 2026Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpoweredcited by 1
  15. 2026Efficient DP-SGD for LLMs with Randomized Clippingcited by 1
  16. 2026Differentially Private Preference Data Synthesis for Large Language Model Alignmentcited by 1
  17. 2025Communication-Efficient and Differentially Private Vertical Federated Learning with Zeroth-Order Optimizationcited by 0
  18. 2025Forward Learning with Differential Privacycited by 0
  19. 2025VAMO: Efficient Zeroth-Order Variance Reduction for SGD with Faster Convergencecited by 0
  20. 2025Privacy Amplification in Differentially Private Zeroth-Order Optimization with Hidden Statescited by 0
  21. 2025On the Inherent Privacy of Zeroth Order Projected Gradient Descentcited by 0
  22. 2025Stabilization of Perturbed Loss Function: Differential Privacy without Gradient Noisecited by 0
  23. 2025Zeroth-Order Sharpness-Aware Learning with Exponential Tiltingcited by 0
  24. 2026Feed m Birds with One Scone: Accelerating Multi-task Gradient Balancing via Bi-level Optimizationcited by 0
  25. 2026Democratising Clinical AI through Dataset Condensation for Classical Clinical Modelscited by 0
  26. 2026New Hybrid Fine-Tuning Paradigm for LLMs: Algorithm Design and Convergence Analysis Frameworkcited by 0
  27. 2026PACZero: PAC-Private Fine-Tuning of Language Models via Sign Quantizationcited by 0
  28. 2026Noise-Aware Shrinkage for Differentially Private Zeroth-Order Fine-Tuning of Large Language Modelscited by 0
  29. 2026Private Direct Preference Optimization for LLM Alignmentcited by 0