mlsanityData to 7 Oct 2026

Differentially Private Decoding in Large Language Models

arXiv 2205.13621 ↗First posted 26 May 2022cs.CL cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

Recent large-scale natural language processing (NLP) systems use a pre-trained Large Language Model (LLM) on massive and diverse corpora as a headstart. In practice, the pre-trained model is adapted to a wide array of tasks via fine-tuning on task-specific datasets. LLMs, while effective, have been shown to memorize instances of training data thereby potentially revealing private information processed during pre-training. The potential leakage might further propagate to the downstream tasks for which LLMs are fine-tuned. On the other hand, privacy-preserving algorithms usually involve retraining from scratch, which is prohibitively expensive for LLMs. In this work, we propose a simple, easy to interpret, and computationally lightweight perturbation mechanism to be applied to an already trained model at the decoding stage. Our perturbation mechanism is model-agnostic and can be used in conjunction with any LLM. We provide theoretical analysis showing that the proposed mechanism is differentially private, and experimental results showing a privacy-utility trade-off.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 25 of its 30 references that are in the map. First posted on arXiv 26 May 2022.

Builds on

25 of 30 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2016Deep Learning with Differential Privacycited by 2,682
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  9. 2019When Does Label Smoothing Help?cited by 985
  10. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  11. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  12. 2017Renyi Differential Privacycited by 643
13 more
  1. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  2. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  3. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  4. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  5. 2017Generalization in Deep Learningcited by 146
  6. 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 102
  7. 2015Generalization in Adaptive Data Analysis and Holdout Reusecited by 76
  8. 2021Membership Inference Attack Susceptibility of Clinical Language Modelscited by 50
  9. 2018Privacy-preserving Predictioncited by 48
  10. 2022Submix: Practical Private Prediction for Large-Scale Language Modelscited by 31
  11. 2021An Efficient DP-SGD Mechanism for Large Scale NLP Modelscited by 27
  12. 2017Privacy Risk in Machine Learning: Analyzing the Connection to Overfittingcited by 11
  13. 2019PAC learning with stable and private predictionscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 7; 2024: 9; 2025: 8; 2026: 6.

34 papers.

Sort:

By area:

  1. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  2. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  3. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 94
  4. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  5. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  6. 2023"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agentscited by 60
  7. 2022AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Modelcited by 57
  8. 2024Is My Data in Your Retrieval Database? Membership Inference Attacks Against Retrieval Augmented Generationcited by 44
  9. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Modelscited by 42
  10. 2023Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuningcited by 40
  11. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 34
  12. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
22 more
  1. 2025Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agentscited by 26
  2. 2024Private prediction for large-scale synthetic text generationcited by 24
  3. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  4. 2023Differentially Private Natural Language Models: Recent Advances and Future Directionscited by 20
  5. 2022Differentially Private Model Compressioncited by 13
  6. 2023Auditing and Generating Synthetic Data with Controllable Trust Trade-offscited by 13
  7. 2024Data-adaptive Differentially Private Prompt Synthesis for In-Context Learningcited by 10
  8. 2025InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacycited by 10
  9. 2025Proactive Privacy Amnesia for Large Language Models: Safeguarding PII with Negligible Impact on Model Utilitycited by 9
  10. 2025Clustering and Median Aggregation Improve Differentially Private Inferencecited by 5
  11. 2025DP-Fusion: Token-Level Differentially Private Inference for Large Language Modelscited by 5
  12. 2024ObfuscaTune: Obfuscated Offsite Fine-tuning and Inference of Proprietary LLMs on Private Datasetscited by 4
  13. 2024Adaptively Private Next-Token Prediction of Large Language Modelscited by 4
  14. 2025Ontology- and LLM-based Data Harmonization for Federated Learning in Healthcarecited by 3
  15. 2025MaskSQL: Safeguarding Privacy for LLM-Based Text-to-SQL via Abstractioncited by 3
  16. 2026Understanding Users' Privacy Reasoning and Behaviors During Chatbot Use to Support Meaningful Agency in Privacycited by 1
  17. 2026Differential Privacy in Generative AI Agents: Analysis and Optimal Tradeoffscited by 1
  18. 2026DP-OPD: Differentially Private On-Policy Distillation for Language Modelscited by 1
  19. 2025How Private is Your Attention? Bridging Privacy with In-Context Learningcited by 0
  20. 2026Barriers to Counterfactual Credit Attribution for Autoregressive Modelscited by 0
  21. 2026Selective Token-Level Cryptographic Redaction for Privacy-Preserving Clinical Deployment of Large Language Modelscited by 0
  22. 2026PANOPTICON: A PII-Based Assemblage of Naturalistic Output Tokens for Investigating Privacy Leakage Within LLM Context Windowcited by 0