mlsanityData to 7 Oct 2026

A Trembling House of Cards? Mapping Adversarial Attacks against Language Agents

arXiv 2402.10196 ↗First posted 15 Feb 2024cs.CL cs.AIcited by 22

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Language agents powered by large language models (LLMs) have seen exploding development. Their capability of using language as a vehicle for thought and communication lends an incredible level of flexibility and versatility. People have quickly capitalized on this capability to connect LLMs to a wide range of external components and environments: databases, tools, the Internet, robotic embodiment, etc. Many believe an unprecedentedly powerful automation technology is emerging. However, new automation technologies come with new safety risks, especially for intricate systems like language agents. There is a surprisingly large gap between the speed and scale of their development and deployment and our understanding of their safety risks. Are we building a house of cards? In this position paper, we present the first systematic effort in mapping adversarial attacks against language agents. We first present a unified conceptual framework for agents with three major components: Perception, Brain, and Action. Under this framework, we present a comprehensive discussion and propose 12 potential attack scenarios against different components of an agent, covering different attack strategies (e.g., input manipulation, adversarial demonstrations, jailbreaking, backdoors). We also draw connections to successful attack strategies previously applied to LLMs. We emphasize the urgency to gain a thorough understanding of language agent risks before their widespread deployment.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 60 of its 95 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

60 of 95 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safety22 Jan
  2. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents18 Jan
  3. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan
  4. 2023Evil Geniuses: Delving into the Safety of LLM-based Agents20 Nov
  5. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 55

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2017Billion-scale similarity search with GPUscited by 2,477
  9. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  10. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  12. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
38 more
  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  2. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  3. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  4. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  5. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  6. 2016Universal adversarial perturbationscited by 774
  7. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  8. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  9. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  10. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  11. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  12. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  13. 2023Larger language models do in-context learning differentlycited by 386
  14. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  15. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  16. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  17. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  18. 2023Self-collaboration Code Generation via ChatGPTcited by 255
  19. 2023Poisoning Language Models During Instruction Tuningcited by 254
  20. 2022TALM: Tool Augmented Language Modelscited by 197
  21. 2022Active Example Selection for In-Context Learningcited by 195
  22. 2020Blind Backdoors in Deep Learning Modelscited by 174
  23. 2019Combating Adversarial Misspellings with Robust Word Recognitioncited by 162
  24. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  25. 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 153
  26. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 153
  27. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
  28. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
  29. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 129
  30. 2023Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMscited by 117
  31. 2023On the Exploitability of Instruction Tuningcited by 110
  32. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 103
  33. 2023Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debatecited by 92
  34. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 74
  35. 2024MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learningcited by 66
  36. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 54
  37. 2022BITE: Textual Backdoor Attacks with Iterative Trigger Injectioncited by 47
  38. 2023Large Language Models can be Guided to Evade AI-Generated Text Detectioncited by 43

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 10; 2026: 5.

22 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  2. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  3. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  4. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  5. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  6. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 80
  7. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 71
  8. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  9. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  10. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  11. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  12. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23
10 more
  1. 2025Les Dissonances: Cross-Tool Harvesting and Polluting in Pool-of-Tools Empowered LLM Agentscited by 19
  2. 2025On the Robustness of GUI Grounding Models Against Image Attackscited by 14
  3. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  4. 2026You Told Me to Do It: Measuring Instructional Text-induced Private Data Leakage in LLM Agentscited by 3
  5. 2026Adaptive Instruction Composition for Automated LLM Red-Teamingcited by 3
  6. 2025Efficient but Vulnerable: Benchmarking and Defending LLM Batch Prompting Attackcited by 1
  7. 2026Assessing Deanonymization Risks with Stylometry-Assisted LLM Agentcited by 1
  8. 2025Defeating Cerberus: Concept-Guided Privacy-Leakage Mitigation in Multimodal Language Modelscited by 0
  9. 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Modelscited by 0
  10. 2026From Expected Harmfulness to Likelihood: A Probabilistic Reformulation of Jailbreaking LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks
  2. 2024Targeting the Core: A Simple and Effective Method to Attack RAG-based Agents via Direct LLM Manipulation
  3. 2025A Survey of Attacks on Large Language Models
  4. 2025AttackVLA: Benchmarking Adversarial and Backdoor Attacks on Vision-Language-Action Models
  5. 2026Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks
  6. 2026Plant, Persist, Trigger: Sleeper Attack on Large Language Model Agents
  7. 2025Adversarial Attacks on Robotic Vision Language Action Models
  8. 2026Jailbreaking Embodied LLMs via Action-level Manipulation