mlsanityData to 14 Oct 2026

DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass

arXiv 2309.06746 ↗First posted 13 Sep 2023cs.CRcited by 67

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Differentially private stochastic gradient descent (DP-SGD) adds noise to gradients in back-propagation, safeguarding training data from privacy leakage, particularly membership inference. It fails to cover (inference-time) threats like embedding inversion and sensitive attribute inference. It is also costly in storage and computation when used to fine-tune large pre-trained language models (LMs). We propose DP-Forward, which directly perturbs embedding matrices in the forward pass of LMs. It satisfies stringent local DP requirements for training and inference data. To instantiate it using the smallest matrix-valued noise, we devise an analytic matrix Gaussian~mechanism (aMGM) by drawing possibly non-i.i.d. noise from a matrix Gaussian distribution. We then investigate perturbing outputs from different hidden (sub-)layers of LMs with aMGM noises. Its utility on three typical tasks almost hits the non-private baseline and outperforms DP-SGD by up to 7.7pp at a moderate privacy level. It saves 3× time and memory costs compared to DP-SGD with the latest high-speed library. It also reduces the average success rates of embedding inversion and sensitive attribute inference by up to 88pp and 41pp, respectively, whereas DP-SGD fails.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 50 of its 95 references that are in the map. First posted on arXiv 13 Sep 2023.

Builds on

50 of 95 references.

References · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2016Layer Normalizationcited by 5,669
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2016Deep Learning with Differential Privacycited by 2,728
  9. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  10. 2020Extracting Training Data from Large Language Modelscited by 2,189
  11. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
38 more
  1. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  2. 2017Renyi Differential Privacycited by 650
  3. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 468
  4. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  5. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  6. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  7. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 383
  8. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 292
  9. 2018A Survey on Deep Learning for Named Entity Recognitioncited by 282
  10. 2018Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoisingcited by 218
  11. 2020Information Leakage in Embedding Modelscited by 203
  12. 2021Numerical Composition of Differential Privacycited by 180
  13. 2020The Lipschitz Constant of Self-Attentioncited by 174
  14. 2018Adversarial Removal of Demographic Attributes from Text Datacited by 170
  15. 2018Amplification by Shuffling: From Local to Central Differential Privacy via Anonymitycited by 164
  16. 2019Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbationscited by 119
  17. 2021Large-Scale Differentially Private BERTcited by 118
  18. 2021Deep Learning with Label Differential Privacycited by 117
  19. 2019Deep Learning with Gaussian Differential Privacycited by 114
  20. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 93
  21. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 90
  22. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 88
  23. 2021Differential Privacy for Text Analytics via Natural Text Sanitizationcited by 77
  24. 2020Differentially Private Representation for NLP: Formal Guarantee and An Empirical Study on Privacy and Fairnesscited by 67
  25. 2021Natural Language Understanding with Privacy-Preserving BERTcited by 55
  26. 2021Antipodes of Label Differential Privacy: PATE and ALIBIcited by 52
  27. 2019Leveraging Hierarchical Representations for Preserving Privacy and Utility in Textcited by 50
  28. 2022Composition of Differential Privacy & Privacy Amplification by Subsamplingcited by 45
  29. 2017A4NT: Author Attribute Anonymity by Adversarial Training of Neural Machine Translationcited by 42
  30. 2021Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networkscited by 39
  31. 2020Towards Differentially Private Text Representationscited by 29
  32. 2021Efficient Differentially Private Secure Aggregation for Federated Learning via Hardness of Learning with Errorscited by 25
  33. 2018SynTF: Synthetic and Differentially Private Term Frequency Vectors for Privacy-Preserving Text Miningcited by 21
  34. 2022Sentence-level Privacy for Document Embeddingscited by 16
  35. 2022Does Label Differential Privacy Prevent Label Inference Attacks?cited by 11
  36. 2018MVG Mechanism: Differential Privacy under Matrix-Valued Querycited by 7
  37. 2021Locally Differentially Private Sparse Vector Aggregationcited by 5
  38. 2021Improved Matrix Gaussian Mechanism for Differential Privacycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 22; 2025: 18; 2026: 21.

67 papers.

Sort:

By area:

  1. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 119
  2. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
  3. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 51
  4. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  5. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 35
  6. 2023How Trustworthy are Open-Source LLMs? An Assessment under Malicious Demonstrations Shows their Vulnerabilitiescited by 33
  7. 2023Split-and-Denoise: Protect large language model inference with local differential privacycited by 32
  8. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  9. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  10. 2024Privacy Preserving Prompt Engineering: A Surveycited by 25
  11. 2024Towards Urban General Intelligence: A Review and Outlook of Urban Foundation Modelscited by 23
  12. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
38 more
  1. 2025Depth Gives a False Sense of Privacy: LLM Internal States Inversioncited by 17
  2. 2024The Fire Thief Is Also the Keeper: Balancing Usability and Privacy in Promptscited by 13
  3. 20241-Diffractor: Efficient and Utility-Preserving Text Obfuscation Leveraging Word-Level Metric Differential Privacycited by 10
  4. 2025OBLIVIATE: Robust and Practical Machine Unlearning for Large Language Modelscited by 10
  5. 2025PRISM: Privacy-Aware Routing for Adaptive Cloud-Edge LLM Inference via Semantic Sketch Collaborationcited by 10
  6. 2024EmojiPrompt: Generative Prompt Obfuscation for Privacy-Preserving Communication with Cloud-based LLMscited by 9
  7. 2024NAP^2: A Benchmark for Naturalness and Privacy-Preserving Text Rewriting by Learning from Humancited by 8
  8. 2024Eguard: Defending LLM Embeddings Against Inversion Attacks via Text Mutual Information Optimizationcited by 8
  9. 2025DATABench: Evaluating Dataset Auditing in Deep Learning from an Adversarial Perspectivecited by 8
  10. 2024Privacy-preserving Fine-tuning of Large Language Models through Flatnesscited by 6
  11. 2025LLM Access Shield: Domain-Specific LLM Framework for Privacy Policy Compliancecited by 6
  12. 2024Delving into Differentially Private Transformercited by 5
  13. 2025Privacy-Preserving LLM Embedding Transmission for End-Cloud Collaborationcited by 5
  14. 2025Cape: Context-Aware Prompt Perturbation Mechanism with Differential Privacycited by 5
  15. 2025VFLAIR-LLM: A Comprehensive Framework and Benchmark for Split Learning of LLMscited by 5
  16. 2025Parameter-Efficient Fine-Tuning with Differential Privacy for Robust Instruction Adaptation in Large Language Modelscited by 5
  17. 2023ConfusionPrompt: Practical Private Inference for Online Large Language Modelscited by 3
  18. 2025A General Pseudonymization Framework for Cloud-Based LLMs: Replacing Privacy Information in Controlled Text Generationcited by 3
  19. 2025Zero-Shot Privacy-Aware Text Rewriting via Iterative Tree Searchcited by 3
  20. 2025MaskSQL: Safeguarding Privacy for LLM-Based Text-to-SQL via Abstractioncited by 3
  21. 2025Membership Inference Attacks on Tokenizers of Large Language Modelscited by 3
  22. 2026SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendationscited by 3
  23. 2024DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Modelscited by 2
  24. 2024P4GCN: Vertical Federated Social Recommendation with Privacy-Preserving Two-Party Graph Convolution Networkcited by 2
  25. 2024NeurIPS 2023 Competition: Privacy Preserving Federated Learning Document VQAcited by 2
  26. 2024Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language modelscited by 2
  27. 2025ISACL: Internal State Analyzer for Copyrighted Training Data Leakagecited by 2
  28. 2026Differentially Private and Communication Efficient Large Language Model Split Inference via Stochastic Quantization and Soft Promptcited by 2
  29. 2026Towards Privacy-Preserving LLM Inference via Covariant Obfuscation (Technical Report)cited by 2
  30. 2026LatentDx: Latent Multi-Agent Communication for Cross-Hospital Rare-Disease Diagnosiscited by 2
  31. 2026From Prompts to Responses: Dual-Sided Data Leakage and Defense in Split Large Language Modelscited by 2
  32. 2024Neural Networks with (Low-Precision) Polynomial Approximations: New Insights and Techniques for Accuracy Improvementcited by 1
  33. 2024A Framework for Cost-Effective and Self-Adaptive LLM Shaking and Recovery Mechanismcited by 1
  34. 2025SoK: Data Minimization in Machine Learningcited by 1
  35. 2025PLRV-O: Advancing Differentially Private Deep Learning via Privacy Loss Random Variable Optimizationcited by 1
  36. 2025Towards Confidential and Efficient LLM Inference with Dual Privacy Protectioncited by 1
  37. 2025PrivTune: Efficient and Privacy-Preserving Fine-Tuning of Large Language Models via Device-Cloud Collaborationcited by 1
  38. 2026Towards Privacy-Preserving Large Language Model: Text-free Inference Through Alignment and Adaptationcited by 1

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021An Efficient DP-SGD Mechanism for Large Scale NLP Models
  2. 2024Differentially Private Learning Needs Better Model Initialization and Self-Distillation
  3. 2025Efficient Differentially Private Fine-Tuning of LLMs via Reinforcement Learning
  4. 2025Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Models
  5. 2022Directional Privacy for Deep Learning
  6. 2024Too Good to be True? Turn Any Model Differentially Private With DP-Weights
  7. 2026Escaping Iterative Parameter-Space Noise: Differentially Private Learning with a Hypernetwork
  8. 2026Privacy Enhanced PEFT: Tensor Train Decomposition Improves Privacy Utility Tradeoffs under DP-SGD