mlsanityData to 7 Oct 2026

Exploring the Limits of Differentially Private Deep Learning with Group-wise Clipping

arXiv 2212.01539 ↗First posted 3 Dec 2022cs.LG stat.MLcited by 52

Not on the map. Why?

Abstract (arXiv)

Differentially private deep learning has recently witnessed advances in computational efficiency and privacy-utility trade-off. We explore whether further improvements along the two axes are possible and provide affirmative answers leveraging two instantiations of group-wise clipping. To reduce the compute time overhead of private learning, we show that per-layer clipping, where the gradient of each neural network layer is clipped separately, allows clipping to be performed in conjunction with backpropagation in differentially private optimization. This results in private learning that is as memory-efficient and almost as fast per training update as non-private learning for many workflows of interest. While per-layer clipping with constant thresholds tends to underperform standard flat clipping, per-layer clipping with adaptive thresholds matches or outperforms flat clipping under given training epoch constraints, hence attaining similar or better task performance within less wall time. To explore the limits of scaling (pretrained) models in differentially private deep learning, we privately fine-tune the 175 billion-parameter GPT-3. We bypass scaling challenges associated with clipping gradients that are distributed across multiple devices with per-device clipping that clips the gradient of each model piece separately on its host device. Privately fine-tuning GPT-3 with per-device clipping achieves a task performance at ε=1 better than what is attainable by non-privately fine-tuning the largest GPT-2 on a summarization task.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 54 of its 69 references that are in the map. First posted on arXiv 3 Dec 2022.

Builds on

54 of 69 references.

References · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2016Wide Residual Networkscited by 3,057
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2016Deep Learning with Differential Privacycited by 2,682
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
38 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2019Deep Leakage from Gradientscited by 1,004
  3. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  4. 2021Membership Inference Attacks From First Principlescited by 839
  5. 2017Renyi Differential Privacycited by 643
  6. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  7. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  8. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  9. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  10. 2020Label-Only Membership Inference Attackscited by 371
  11. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  12. 2017Deep Models Under the GAN: Information Leakage from Collaborative Deep Learningcited by 326
  13. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  14. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  15. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  16. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  17. 2021Numerical Composition of Differential Privacycited by 174
  18. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 144
  19. 2022Reconstructing Training Data with Informed Adversariescited by 143
  20. 2019Privacy Risks of Securing Machine Learning Models against Adversarial Examplescited by 136
  21. 2021Large-Scale Differentially Private BERTcited by 117
  22. 2020Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identificationcited by 88
  23. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  24. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  25. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  26. 2018A General Approach to Adding Differential Privacy to Iterative Training Procedurescited by 84
  27. 2019AdaCliP: Adaptive Clipping for Private SGDcited by 77
  28. 2020Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorizationcited by 58
  29. 2022Mixed Differential Privacy in Computer Visioncited by 53
  30. 2022When Does Differentially Private Learning Not Suffer in High Dimensions?cited by 48
  31. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
  32. 2020Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clippingcited by 46
  33. 2020Differentially Private Language Models Benefit from Public Pre-trainingcited by 46
  34. 2021Private Adaptive Gradient Methods for Convex Optimizationcited by 46
  35. 2015Efficient Per-Example Gradient Computationscited by 42
  36. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  37. 2021Fast and Memory Efficient Differentially Private-SGD via JL Projectionscited by 29
  38. 2021Benchmarking Differential Privacy and Federated Learning for BERT Modelscited by 28

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 16; 2024: 18; 2025: 11; 2026: 7.

52 papers.

Sort:

By area:

  1. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 74
  2. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 63
  3. 2023Privacy-Preserving In-Context Learning for Large Language Modelscited by 50
  4. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  5. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  6. 2023Why Is Public Pretraining Necessary for Private Model Training?cited by 36
  7. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 34
  8. 2025Towards Label-Only Membership Inference Attack against Pre-trained Large Language Modelscited by 34
  9. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  10. 2024Learnable Privacy Neurons Localization in Language Modelscited by 26
  11. 2023Correlated Noise Provably Beats Independent Noise for Differentially Private Learningcited by 23
  12. 2023Selective Pre-training for Private Fine-tuningcited by 22
38 more
  1. 2024How Private are DP-SGD Implementations?cited by 22
  2. 2025Privacy Auditing of Large Language Modelscited by 22
  3. 2023Differentially Private Image Classification by Learning Priors from Random Processescited by 19
  4. 2023Unlocking Accuracy and Fairness in Differentially Private Image Classificationcited by 16
  5. 2024Scalable DP-SGD: Shuffling vs. Poisson Subsamplingcited by 16
  6. 2024Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuningcited by 14
  7. 2023TMI! Finetuned Models Leak Private Information from their Pretraining Datacited by 13
  8. 2024Balls-and-Bins Sampling for DP-SGDcited by 13
  9. 2025Differentially Private Synthetic Data via APIs 3: Using Simulators Instead of Foundation Modelcited by 13
  10. 2023DP-SGD Without Clipping: The Lipschitz Neural Network Waycited by 11
  11. 2024Data-adaptive Differentially Private Prompt Synthesis for In-Context Learningcited by 10
  12. 2024LLM-based Privacy Data Augmentation Guided by Knowledge Distillation with a Distribution Tutor for Medical Text Classificationcited by 8
  13. 2024Pre-training Differentially Private Models with Limited Public Datacited by 8
  14. 2025Synthesize Privacy-Preserving High-Resolution Images via Private Textual Intermediariescited by 8
  15. 2023On the accuracy and efficiency of group-wise clipping in differentially private optimizationcited by 7
  16. 2023Zero redundancy distributed learning with differential privacycited by 7
  17. 2023Optimal Unbiased Randomizers for Regression with Label Differential Privacycited by 6
  18. 2025Automated Privacy Information Annotation in Large Language Model Interactionscited by 6
  19. 2023On the Benefits of Public Representations for Private Transfer Learning under Distribution Shiftcited by 5
  20. 2024Delving into Differentially Private Transformercited by 5
  21. 2026Fundamental Limitations of Favorable Privacy-Utility Guarantees for DP-SGDcited by 4
  22. 2023DPFormer: Learning Differentially Private Transformer on Long-Tailed Datacited by 2
  23. 2024Differentially Private Training of Mixture of Experts Modelscited by 2
  24. 2024Quantifying Memorization and Detecting Training Data of Pre-trained Language Models using Japanese Newspapercited by 2
  25. 2024Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clippingcited by 2
  26. 2024Differentially Private Prototypes for Imbalanced Transfer Learningcited by 2
  27. 2025FlashDP: Private Training Large Language Models with Efficient DP-SGDcited by 1
  28. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  29. 2026Q-ShiftDP: A Differentially Private Parameter-Shift Rule for Quantum Machine Learningcited by 1
  30. 2026Differentially Private Preference Data Synthesis for Large Language Model Alignmentcited by 1
  31. 2024LMO-DP: Optimizing the Randomization Mechanism for Differentially Private Fine-Tuning (Large) Language Modelscited by 0
  32. 2024Weights Shuffling for Improving DPSGD in Transformer-based Modelscited by 0
  33. 2025Private Training & Data Generation by Clustering Embeddingscited by 0
  34. 2025Lower Bounds for Public-Private Learning under Distribution Shiftcited by 0
  35. 2025Stabilization of Perturbed Loss Function: Differential Privacy without Gradient Noisecited by 0
  36. 2025Rethinking Layer-wise Gaussian Noise Injection: Bridging Implicit Objectives and Privacy Budget Allocationcited by 0
  37. 2026Sequential Subspace Noise Injection Prevents Accuracy Collapse in Certified Unlearningcited by 0
  38. 2026Mitigating Membership Inference in Intermediate Representations with Differentially Private Trainingcited by 0

50 most cited of 52 All 52 (JSON)