mlsanityData to 7 Oct 2026

Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks

arXiv 2310.10844 ↗First posted 16 Oct 2023cs.CL cs.CR cs.LGcited by 189

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs) are swiftly advancing in architecture and capability, and as they integrate more deeply into complex systems, the urgency to scrutinize their security properties grows. This paper surveys research in the emerging interdisciplinary field of adversarial attacks on LLMs, a subfield of trustworthy ML, combining the perspectives of Natural Language Processing and Security. Prior work has shown that even safety-aligned LLMs (via instruction tuning and reinforcement learning through human feedback) can be susceptible to adversarial attacks, which exploit weaknesses and mislead AI systems, as evidenced by the prevalence of `jailbreak' attacks on models like ChatGPT and Bard. In this survey, we first provide an overview of large language models, describe their safety alignment, and categorize existing research based on various learning structures: textual-only attacks, multi-modal attacks, and additional attack methods specifically targeting complex systems, such as federated learning or multi-agent systems. We also offer comprehensive remarks on works that focus on the fundamental sources of vulnerabilities and potential defenses. To make this field more accessible to newcomers, we present a systematic review of existing works, a structured typology of adversarial attack concepts, and additional resources, including slides for presentations on related topics at the 62nd Annual Meeting of the Association for Computational Linguistics (ACL'24).

Key facts. Cited by 189 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 109 of its 182 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

109 of 182 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 108

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
38 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2016Deep Learning with Differential Privacycited by 2,682
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,911
  13. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  14. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  15. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  16. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  17. 2017Random Erasing Data Augmentationcited by 1,235
  18. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  19. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  20. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  21. 2022Red Teaming Language Models with Language Modelscited by 983
  22. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  23. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  24. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  25. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 739
  26. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  27. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  28. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  29. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  30. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  31. 2017Provable defenses against adversarial examples via the convex outer adversarial polytopecited by 564
  32. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  33. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  34. 2023LERF: Language Embedded Radiance Fieldscited by 538
  35. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  36. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 479
  37. 2018Certified Robustness to Adversarial Examples with Differential Privacycited by 462
  38. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435

50 most cited of 108 All 108 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 81; 2025: 73; 2026: 28.

189 papers.

Sort:

By area:

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMscited by 424
  3. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  4. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  5. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  6. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  7. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  8. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  9. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  10. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  11. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 113
  12. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
38 more
  1. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  2. 2023A Survey of Graph Meets Large Language Model: Progress and Future Directionscited by 84
  3. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  4. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  5. 2024A New Era in LLM Security: Exploring Security Concerns in Real-World LLM-based Systemscited by 74
  6. 2024Safeguarding Large Language Models: A Surveycited by 73
  7. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  8. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  9. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuningcited by 63
  10. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 61
  11. 2024White-box Multimodal Jailbreaks Against Large Vision-Language Modelscited by 57
  12. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Timecited by 57
  13. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  14. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  15. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  16. 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 41
  17. 2024Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Promptscited by 39
  18. 2024A Cross-Language Investigation into Jailbreak Attacks in Large Language Modelscited by 37
  19. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  20. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  21. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  22. 2024Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Surveycited by 34
  23. 2024IDEATOR: Jailbreaking and Benchmarking Large Vision-Language Models Using Themselvescited by 34
  24. 2024SPML: A DSL for Defending Language Models Against Prompt Attackscited by 33
  25. 2024Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Modelscited by 33
  26. 2024All in How You Ask for It: Simple Black-Box Method for Jailbreak Attackscited by 30
  27. 2025Security Concerns for Large Language Models: A Surveycited by 26
  28. 2024Security of AI Agentscited by 24
  29. 2024What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attackscited by 24
  30. 2024"Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreakcited by 23
  31. 2024Goal-guided Generative Prompt Injection Attack on Large Language Modelscited by 22
  32. 2024Cross-Modal Safety Alignment: Is textual unlearning all you need?cited by 22
  33. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 21
  34. 2024Breaking ReAct Agents: Foot-in-the-Door Attack Will Get You Incited by 21
  35. 2024Sandwich attack: Multi-language Mixture Adaptive Attack on LLMscited by 20
  36. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveyscited by 20
  37. 2024RapGuard: Safeguarding Multimodal Large Language Models via Rationale-aware Defensive Promptingcited by 19
  38. 2024Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Securitycited by 18

50 most cited of 189 All 189 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Attacks on Large Language Models
  2. 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers
  3. 2025Data Security in Large Language Models: Risks, Defense, and Directions
  4. 2024Large Language Model Safety: A Holistic Survey
  5. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  6. 2025Scaling Patterns in Adversarial Alignment: Evidence from Multi-LLM Jailbreak Experiments
  7. 2025Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMs
  8. 2025Adversarial Reasoning at Jailbreaking Time