mlsanityData to 14 Oct 2026

Privacy in Large Language Models: Attacks, Defenses and Future Directions

arXiv 2310.10383 ↗First posted 16 Oct 2023cs.CL cs.CRcited by 51

Not on the map. Why?

Abstract (arXiv)

The advancement of large language models (LLMs) has significantly enhanced the ability to effectively tackle various downstream NLP tasks and unify these tasks into generative pipelines. On the one hand, powerful language models, trained on massive textual data, have brought unparalleled accessibility and usability for both models and users. On the other hand, unrestricted access to these models can also introduce potential malicious and unintentional privacy risks. Despite ongoing efforts to address the safety and privacy concerns associated with LLMs, the problem remains unresolved. In this paper, we provide a comprehensive analysis of the current privacy attacks targeting LLMs and categorize them according to the adversary's assumed capabilities to shed light on the potential vulnerabilities present in LLMs. Then, we present a detailed overview of prominent defense strategies that have been developed to counter these privacy attacks. Beyond existing works, we identify upcoming privacy concerns as LLMs evolve. Lastly, we point out several potential avenues for future exploration.

Key facts. Cited by 51 papers in the map (data through 14 Oct 2026). Builds on 172 of its 313 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

172 of 313 references.

References · 172

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  11. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  12. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
38 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  3. 2017Deep reinforcement learning from human preferencescited by 3,912
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2016Deep Learning with Differential Privacycited by 2,728
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2020Extracting Training Data from Large Language Modelscited by 2,189
  13. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  14. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  15. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  16. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  17. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  18. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  19. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  20. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  21. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  22. 2022Red Teaming Language Models with Language Modelscited by 1,026
  23. 2019Deep Leakage from Gradientscited by 1,021
  24. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  25. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  26. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  27. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  28. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  29. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  30. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 580
  31. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  32. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  33. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  34. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  35. 2018Detecting Backdoor Attacks on Deep Neural Networks by Activation Clusteringcited by 456
  36. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  37. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  38. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431

50 most cited of 172 All 172 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 24; 2025: 18; 2026: 9.

51 papers.

Sort:

By area:

  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  2. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  3. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  4. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
  5. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  6. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 81
  7. 2024Safeguarding Large Language Models: A Surveycited by 78
  8. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
  9. 2024Building Guardrails for Large Language Modelscited by 65
  10. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 61
  11. 2024Large Language Model Safety: A Holistic Surveycited by 59
  12. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 43
38 more
  1. 2024When Large Language Models Meet Vector Databases: A Surveycited by 39
  2. 2024IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselvescited by 37
  3. 2024Large Language Models in Cybersecurity: State-of-the-Artcited by 34
  4. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  5. 2024Evaluating and Enhancing LLMs Agent based on Theory of Mind in Guandan: A Multi-Player Cooperative Game under Imperfect Informationcited by 20
  6. 20251-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoningcited by 17
  7. 2024Differential Privacy Mechanisms in Neural Tangent Kernel Regressioncited by 14
  8. 2024Provable Differentially Private Computation of the Cross-Attention Mechanismcited by 14
  9. 2024Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directionscited by 14
  10. 2024EventGround: Narrative Reasoning by Grounding to Eventuality-centric Knowledge Graphscited by 9
  11. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  12. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  13. 2024Constrained Reasoning Chains for Enhancing Theory-of-Mind in Large Language Modelscited by 5
  14. 2025Machine Learners Should Acknowledge the Legal Implications of Large Language Models as Personal Datacited by 5
  15. 2026TrinityGuard: A Unified Framework for Safeguarding Multi-Agent Systemscited by 5
  16. 2024AIDBench: A benchmark for evaluating the authorship identification capability of large language modelscited by 4
  17. 2025Big Help or Big Brother? Auditing Tracking, Profiling, and Personalization in Generative AI Assistantscited by 4
  18. 2025Simulated Ensemble Attack: Transferring Jailbreaks Across Fine-tuned Vision-Language Modelscited by 4
  19. 2026Positive-Unlabeled Reinforcement Learning Distillation for On-Premise Small Modelscited by 4
  20. 2025Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Modelscited by 3
  21. 2025HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoringcited by 3
  22. 2025Backdoor-Powered Prompt Injection Attacks Nullify Defense Methodscited by 3
  23. 2026SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendationscited by 3
  24. 2025Distributional Statistics Restore Training Data Auditability in One-step Distilled Diffusion Modelscited by 2
  25. 2025Safety Compliance: Rethinking LLM Safety Reasoning through the Lens of Compliancecited by 2
  26. 2025A Longitudinal Measurement of Privacy Policy Evolution for Large Language Modelscited by 2
  27. 2025SoDA: An Efficient Interaction Paradigm for the Agentic Webcited by 2
  28. 2025Towards Provable (In)Secure Model Weight Release Schemescited by 1
  29. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  30. 2025Sentra-Guard: A Real-Time Multilingual Defense Against Adversarial LLM Promptscited by 1
  31. 2026The Alignment Curse: Modality Alignment Supercharges Audio Attacks via Text Transfercited by 1
  32. 2026MultivationBench: A Benchmark for Multimodal Sequential Motivation Reasoningcited by 1
  33. 2024Fortifying Ethical Boundaries in AI: Advanced Strategies for Enhancing Security in Large Language Modelscited by 0
  34. 2024Exploring the Privacy Protection Capabilities of Chinese Large Language Modelscited by 0
  35. 2025Quantifying Return on Security Controls in LLM Systemscited by 0
  36. 2026Combating Data Laundering in LLM Trainingcited by 0
  37. 2026COMPASS: COntinual Multilingual PEFT with Adaptive Semantic Samplingcited by 0
  38. 2026On the Privacy of LLMs: An Ablation Studycited by 0

50 most cited of 51 All 51 (JSON)