mlsanityData to 7 Oct 2026

Differentially Private Bias-Term Fine-tuning of Foundation Models

arXiv 2210.00036 ↗First posted 30 Sep 2022cs.LG cs.CL cs.CRcited by 47

Not on the map. Why?

Abstract (arXiv)

We study the problem of differentially private (DP) fine-tuning of large pre-trained models -- a recent privacy-preserving approach suitable for solving downstream tasks with sensitive data. Existing work has demonstrated that high accuracy is possible under strong privacy constraint, yet requires significant computational overhead or modifications to the network architecture. We propose differentially private bias-term fine-tuning (DP-BiTFiT), which matches the state-of-the-art accuracy for DP algorithms and the efficiency of the standard BiTFiT. DP-BiTFiT is model agnostic (not modifying the network architecture), parameter efficient (only training about 0.1% of the parameters), and computation efficient (almost removing the overhead caused by DP, in both the time and space complexity). On a wide range of tasks, DP-BiTFiT is 2~30X faster and uses 2~8X less memory than DP full fine-tuning, even faster than the standard full fine-tuning. This amazing efficiency enables us to conduct DP fine-tuning on language and vision tasks with long-sequence texts and high-resolution images, which were computationally difficult using existing methods. We open-source our code at FastDP (https://github.com/awslabs/fast-differential-privacy).

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 51 of its 70 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

51 of 70 references.

References · 51

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2020Longformer: The Long-Document Transformercited by 3,362
38 more
  1. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  2. 2016Deep Learning with Differential Privacycited by 2,682
  3. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  7. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  8. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  9. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  10. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  11. 2021GPT Understands, Toocited by 715
  12. 2018Do Better ImageNet Models Transfer Better?cited by 560
  13. 2017Learning multiple visual domains with residual adapterscited by 553
  14. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  15. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  16. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  17. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  18. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  19. 2018Subsampled R'enyi Differential Privacy and Analytical Moments Accountantcited by 234
  20. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  21. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  22. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  23. 2019R'enyi Differential Privacy of the Sampled Gaussian Mechanismcited by 199
  24. 2021Numerical Composition of Differential Privacycited by 174
  25. 2020Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learningcited by 139
  26. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  27. 2019Deep Learning with Gaussian Differential Privacycited by 112
  28. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 90
  29. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  30. 2022Toward Training at ImageNet Scale with Differential Privacycited by 87
  31. 2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
  32. 2019Evaluating the State-of-the-Art of End-to-End Natural Language Generation: The E2E NLG Challengecited by 86
  33. 2021Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learningcited by 86
  34. 2020Enabling Fast Differentially Private SGD via Just-in-Time Compilation and Vectorizationcited by 58
  35. 2022Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacycited by 47
  36. 2020Scaling up Differentially Private Deep Learning with Fast Per-Example Gradient Clippingcited by 46
  37. 2015Efficient Per-Example Gradient Computationscited by 42
  38. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 12; 2024: 14; 2025: 13; 2026: 5.

47 papers.

Sort:

By area:

  1. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 74
  2. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  3. 2023Privacy-Preserving In-Context Learning for Large Language Modelscited by 50
  4. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  5. 2023Why Is Public Pretraining Necessary for Private Model Training?cited by 36
  6. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 34
  7. 2023Can Public Large Language Models Help Private Cross-device Federated Learning?cited by 30
  8. 2023Integration of Large Language Models and Federated Learningcited by 28
  9. 2023Selective Pre-training for Private Fine-tuningcited by 22
  10. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 21
  11. 2023Differentially Private Natural Language Models: Recent Advances and Future Directionscited by 20
  12. 2025Differentially Private Steering for Large Language Model Alignmentcited by 15
35 more
  1. 2023Choosing Public Datasets for Private Machine Learning via Gradient Subspace Distancecited by 14
  2. 2023TMI! Finetuned Models Leak Private Information from their Pretraining Datacited by 13
  3. 2023Tangent Transformers for Composition, Privacy and Removalcited by 13
  4. 2022A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimizationcited by 11
  5. 2024DP-TabICL: In-Context Learning with Differentially Private Tabular Datacited by 11
  6. 2024Pre-training Differentially Private Models with Limited Public Datacited by 8
  7. 2024Neural Collapse Meets Differential Privacy: Curious Behaviors of NoisyGD with Near-perfect Representation Learningcited by 8
  8. 2025Synthesize Privacy-Preserving High-Resolution Images via Private Textual Intermediariescited by 8
  9. 2022Differentially Private Image Classification from Featurescited by 7
  10. 2023On the accuracy and efficiency of group-wise clipping in differentially private optimizationcited by 7
  11. 2023Zero redundancy distributed learning with differential privacycited by 7
  12. 2025Towards hyperparameter-free optimization with differential privacycited by 7
  13. 2022Differentially Private Optimizers Can Learn Adversarially Robust Modelscited by 6
  14. 2023Privacy and Copyright Protection in Generative AI: A Lifecycle Perspectivecited by 6
  15. 2024Privacy-preserving Fine-tuning of Large Language Models through Flatnesscited by 6
  16. 2024Public-data Assisted Private Stochastic Optimization: Power and Limitationscited by 5
  17. 2025Parameter-Efficient Fine-Tuning with Differential Privacy for Robust Instruction Adaptation in Large Language Modelscited by 5
  18. 2025Can Differentially Private Fine-tuning LLMs Protect Against Privacy Attacks?cited by 4
  19. 2025SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRAcited by 4
  20. 2025A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challengescited by 3
  21. 2024Training Large ASR Encoders with Differential Privacycited by 2
  22. 2024Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language modelscited by 2
  23. 2025Towards Privacy-aware Mental Health AI Models: Advances, Challenges, and Opportunitiescited by 2
  24. 2025PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHingcited by 2
  25. 2024Dataset Condensation Driven Machine Unlearningcited by 1
  26. 2025Tuning without Peeking: Provable Generalization Bounds and Robust LLM Post-Trainingcited by 1
  27. 2025DP-MicroAdam: Private and Frugal Algorithm for Training and Fine-tuningcited by 1
  28. 2026FIBER: A Differentially Private Optimizer with Filter-Aware Innovation Bias Correctioncited by 1
  29. 2024LMO-DP: Optimizing the Randomization Mechanism for Differentially Private Fine-Tuning (Large) Language Modelscited by 0
  30. 2025Towards Reliable and Generalizable Differentially Private Machine Learning (Extended Version)cited by 0
  31. 2025Re-examining Low Rank adaptation for private LLM fine-tuningcited by 0
  32. 2026Lap2: Revisiting Laplace DP-SGD for High Dimensions via Majorization Theorycited by 0
  33. 2026Spectral Gradient Orthogonalization Improves Differentially Private Training at Scalecited by 0
  34. 2026When Privacy Hurts Mergeability: Geometry-Aware Model Merging under Differential Privacycited by 0
  35. 2026Only Pay What You Must Spend: On-Demand Privacy Budget Payment for Differentially Private RAGcited by 0