mlsanityData to 14 Oct 2026

A Survey of Attacks on Large Language Models

arXiv 2505.12567 ↗First posted 18 May 2025cs.CR cs.AIcited by 15

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large language models (LLMs) and LLM-based agents have been widely deployed in a wide range of applications in the real world, including healthcare diagnostics, financial analysis, customer support, robotics, and autonomous driving, expanding their powerful capability of understanding, reasoning, and generating natural languages. However, the wide deployment of LLM-based applications exposes critical security and reliability risks, such as the potential for malicious misuse, privacy leakage, and service disruption that weaken user trust and undermine societal safety. This paper provides a systematic overview of the details of adversarial attacks targeting both LLMs and LLM-based agents. These attacks are organized into three phases in LLMs: Training-Phase Attacks, Inference-Phase Attacks, and Availability & Integrity Attacks. For each phase, we analyze the details of representative and recently introduced attack methods along with their corresponding defenses. We hope our survey will provide a good tutorial and a comprehensive understanding of LLM security, especially for attacks on LLMs. We desire to raise attention to the risks inherent in widely deployed LLM-based applications and highlight the urgent need for robust mitigation strategies for evolving threats.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 68 of its 87 references that are in the map. First posted on arXiv 18 May 2025.

Builds on

68 of 87 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM-based Agent18 Feb
  2. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems9 Oct
  3. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents5 Jun

Other references · 65

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  10. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  11. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  12. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
38 more
  1. 2023A Comprehensive Overview of Large Language Modelscited by 648
  2. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  3. 2023Textbooks Are All You Needcited by 569
  4. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
  5. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  6. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  7. 2023Detecting Language Model Attacks with Perplexitycited by 431
  8. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 429
  9. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  10. 2023Can AI-Generated Text be Reliably Detected?cited by 386
  11. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  12. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  13. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
  14. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 268
  15. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 267
  16. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 267
  17. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 263
  18. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 240
  19. 2023Undetectable Watermarks for Language Modelscited by 226
  20. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 222
  21. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  22. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  23. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 157
  24. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 143
  25. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  26. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  27. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 119
  28. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 107
  29. 2024BadEdit: Backdooring large language models by model editingcited by 96
  30. 2021Hidden Backdoors in Human-Centric Language Modelscited by 90
  31. 2018Scenic: A Language for Scenario Specification and Scene Generationcited by 83
  32. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 80
  33. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 76
  34. 2023Composite Backdoor Attacks Against Large Language Modelscited by 73
  35. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Cluescited by 68
  36. 2025Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Modelscited by 64
  37. 2024Instruction Backdoor Attacks Against Customized LLMscited by 62
  38. 2023Red Teaming Language Model Detectors with Language Modelscited by 55

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2026Beyond Max Tokens: Stealthy Resource Amplification via Tool Calling Chains in LLM Agentscited by 14
  3. 2025Exploiting Web Search Tools of AI Agents for Data Exfiltrationcited by 3
  4. 2025Countermind: A Multi-Layered Security Architecture for Large Language Modelscited by 2
  5. 2026Efficient Multi-objective Prompt Optimization via Pure-exploration Banditscited by 2
  6. 2025Lumos: Let there be Language Model System Certificationcited by 1
  7. 2026Are LLMs Vulnerable to Preference-Undermining Attacks (PUA)? A Factorial Analysis Methodology for Diagnosing the Trade-off between Preference Alignment and Real-World Validitycited by 1
  8. 2026Step-Wise Refusal Dynamics in Autoregressive and Diffusion Language Modelscited by 1
  9. 2026Prompt Control-Flow Integrity: A Priority-Aware Runtime Defense Against Prompt Injection in LLM Systemscited by 1
  10. 2026Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attackscited by 1
  11. 2025Defending Large Language Models Against Jailbreak Exploits with Responsible AI Considerationscited by 0
  12. 2026Why Do Large Language Models Generate Harmful Content?cited by 0
3 more
  1. 2026A Lifecycle and Application-Stack Survey of Large Language Model Vulnerabilities: Attacks, Risks, Defenses, and Open Problemscited by 0
  2. 2026The Role of Rigor in Artificial Intelligencecited by 0
  3. 2026StableJudge: A Lightweight Adversarial Defense Framework for LLM-as-a-Judgecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Breaking Down the Defenses: A Comparative Survey of Attacks on Large Language Models
  2. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  3. 2024Recent Advances in Attack and Defense Approaches of Large Language Models
  4. 2025Data Security in Large Language Models: Risks, Defense, and Directions
  5. 2025Attack and defense techniques in large language models: A survey and new perspectives
  6. 2024Unique Security and Privacy Threats of Large Language Models: A Comprehensive Survey
  7. 2024Exploring Vulnerabilities and Protections in Large Language Models: A Survey
  8. 2024Large Language Model Safety: A Holistic Survey