mlsanityData to 7 Oct 2026

Security and Privacy Challenges of Large Language Models: A Survey

arXiv 2402.00888 ↗First posted 30 Jan 2024cs.CL cs.AI cs.CRcited by 179

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated extraordinary capabilities and contributed to multiple fields, such as generating and summarizing text, language translation, and question-answering. Nowadays, LLM is becoming a very popular tool in computerized language processing tasks, with the capability to analyze complicated linguistic patterns and provide relevant and appropriate responses depending on the context. While offering significant advantages, these models are also vulnerable to security and privacy attacks, such as jailbreaking attacks, data poisoning attacks, and Personally Identifiable Information (PII) leakage attacks. This survey provides a thorough review of the security and privacy challenges of LLMs for both training data and users, along with the application-based risks in various domains, such as transportation, education, and healthcare. We assess the extent of LLM vulnerabilities, investigate emerging security and privacy attacks for LLMs, and review the potential defense mechanisms. Additionally, the survey outlines existing research gaps in this domain and highlights future research directions.

Key facts. Cited by 179 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 165 of its 293 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

165 of 293 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?3 Aug
  2. 2023From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy3 Jul
  3. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  4. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPT11 Apr

Other references · 161

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  11. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  12. 2021Evaluating Large Language Models Trained on Codecited by 8,053
38 more
  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  2. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  3. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2016Deep Learning with Differential Privacycited by 2,682
  10. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
  13. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  14. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  15. 2019Language Models as Knowledge Bases?cited by 1,922
  16. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  17. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  18. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  19. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  20. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  21. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  22. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  23. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  24. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  25. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  26. 2019Deep Leakage from Gradientscited by 1,004
  27. 2021Ethical and social risks of harm from Language Modelscited by 957
  28. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  29. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  30. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  31. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  32. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  33. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  34. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  35. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  36. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  37. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  38. 2018Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networkscited by 550

50 most cited of 161 All 161 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 38; 2025: 92; 2026: 49.

179 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2025Small Language Models are the Future of Agentic AIcited by 254
  3. 2024Tool Learning with Large Language Models: A Surveycited by 221
  4. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  5. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  6. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  7. 2024Clio: Privacy-Preserving Insights into Real-World AI Usecited by 66
  8. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 65
  9. 2024PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensemblescited by 54
  10. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  11. 2024HelloBench: Evaluating Long Text Generation Capabilities of Large Language Modelscited by 42
  12. 2024A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directionscited by 34
38 more
  1. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  2. 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 29
  3. 2024A Survey of Useful LLM Evaluationcited by 28
  4. 2025A Comprehensive Survey on the Trustworthiness of Large Language Models in Healthcarecited by 27
  5. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  6. 2025Exploring the Roles of Large Language Models in Reshaping Transportation Systems: A Survey, Framework, and Roadmapcited by 26
  7. 2025Security Concerns for Large Language Models: A Surveycited by 26
  8. 2024CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaborationcited by 25
  9. 2024Privacy Preserving Prompt Engineering: A Surveycited by 24
  10. 2024"Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreakcited by 23
  11. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  12. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 21
  13. 2025ASIDE: Architectural Separation of Instructions and Data in Language Modelscited by 21
  14. 2025How malicious AI swarms can threaten democracy: The fusion of agentic AI and LLMs marks a new frontier in information warfarecited by 20
  15. 2024Causal Evaluation of Language Modelscited by 18
  16. 2025A Survey on Model Extraction Attacks and Defenses for Large Language Modelscited by 18
  17. 2025Membership Inference Attacks on Large-Scale Models: A Surveycited by 17
  18. 2025Security of Internet of Agents: Attacks and Countermeasurescited by 17
  19. 2025GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detectioncited by 17
  20. 2024HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Routercited by 16
  21. 2025Large Language Models for Computer-Aided Design: A Surveycited by 16
  22. 2024DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Modelscited by 15
  23. 2024Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysiscited by 14
  24. 2025Doxing via the Lens: Revealing Location-related Privacy Leakage on Multi-modal Large Reasoning Modelscited by 14
  25. 2025To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Promptcited by 14
  26. 2025AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learningcited by 14
  27. 2025"Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editorscited by 14
  28. 2026Security Threat Modeling for Emerging AI-Agent Protocols: A Comparative Analysis of MCP, A2A, Agora, and ANPcited by 14
  29. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  30. 2025A Survey of Attacks on Large Language Modelscited by 13
  31. 2025Editing as Unlearning: Are Knowledge Editing Methods Strong Baselines for Large Language Model Unlearning?cited by 13
  32. 2025LLM-Enabled In-Context Learning for Data Collection Scheduling in UAV-assisted Sensor Networkscited by 12
  33. 2025Large Reasoning Models Are Autonomous Jailbreak Agentscited by 12
  34. 2025GRU: Mitigating the Trade-off between Unlearning and Retention for LLMscited by 11
  35. 2025BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Textcited by 11
  36. 2025The Trust Paradox in LLM-Based Multi-Agent Systems: When Collaboration Becomes a Security Vulnerabilitycited by 11
  37. 2024Exploring Vulnerabilities and Protections in Large Language Models: A Surveycited by 10
  38. 2024GenBFA: An Evolutionary Optimization Approach to Bit-Flip Attacks on LLMscited by 10

50 most cited of 179 All 179 (JSON)