mlsanityData to 7 Oct 2026

Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents

arXiv 2402.11208 ↗First posted 17 Feb 2024cs.CR cs.AI cs.CLcited by 135

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Driven by the rapid development of Large Language Models (LLMs), LLM-based agents have been developed to handle various real-world applications, including finance, healthcare, and shopping, etc. It is crucial to ensure the reliability and security of LLM-based agents during applications. However, the safety issues of LLM-based agents are currently under-explored. In this work, we take the first step to investigate one of the typical safety threats, backdoor attack, to LLM-based agents. We first formulate a general framework of agent backdoor attacks, then we present a thorough analysis of different forms of agent backdoor attacks. Specifically, compared with traditional backdoor attacks on LLMs that are only able to manipulate the user inputs and model outputs, agent backdoor attacks exhibit more diverse and covert forms: (1) From the perspective of the final attacking outcomes, the agent backdoor attacker can not only choose to manipulate the final output distribution, but also introduce the malicious behavior in an intermediate reasoning step only, while keeping the final output correct. (2) Furthermore, the former category can be divided into two subcategories based on trigger locations, in which the backdoor trigger can either be hidden in the user query or appear in an intermediate observation returned by the external environment. We implement the above variations of agent backdoor attacks on two typical agent tasks including web shopping and tool utilization. Extensive experiments show that LLM-based agents suffer severely from backdoor attacks and such backdoor vulnerability cannot be easily mitigated by current textual backdoor defense algorithms. This indicates an urgent need for further research on the development of targeted defenses against backdoor attacks on LLM-based agents. Warning: This paper may contain biased content.

Key facts. Cited by 135 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 52 of its 71 references that are in the map. First posted on arXiv 17 Feb 2024.

Builds on

52 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training10 Jan
  2. 2023Evil Geniuses: Delving into the Safety of LLM-based Agents20 Nov

Other references · 50

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
38 more
  1. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  3. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  5. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  6. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  8. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  9. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  11. 2022PAL: Program-aided Language Modelscited by 635
  12. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  13. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  14. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  15. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  16. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  17. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  18. 2023Tool Learning with Foundation Modelscited by 302
  19. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  20. 2023Poisoning Language Models During Instruction Tuningcited by 254
  21. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 203
  22. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
  23. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  24. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 100
  25. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Modelscited by 97
  26. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Modelscited by 96
  27. 2021Backdoor Pre-trained Models Can Transfer to Allcited by 79
  28. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 74
  29. 2021BACKDOORL: Backdoor Attack against Competitive Reinforcement Learningcited by 66
  30. 2023Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermarkcited by 61
  31. 2022Fine-mixing: Mitigating Backdoors in Fine-tuned Language Modelscited by 50
  32. 2023Stealthy and Persistent Unalignment on Large Language Models via Backdoor Injectionscited by 37
  33. 2021Provably Efficient Black-Box Action Poisoning Attacks Against Reinforcement Learningcited by 29
  34. 2023BadRL: Sparse Targeted Backdoor Attack Against Reinforcement Learningcited by 29
  35. 2022BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasetscited by 24
  36. 2023Defending against Insertion-based Textual Backdoor Attacks via Attributioncited by 24
  37. 2022Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attackscited by 18
  38. 2022A Temporal-Pattern Backdoor Attack to Deep Reinforcement Learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 61; 2026: 45.

135 papers.

Sort:

By area:

  1. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  2. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  3. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
  4. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  5. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  6. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  7. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  8. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  9. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  10. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 112
  11. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  12. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
38 more
  1. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilitiescited by 90
  2. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  3. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 80
  4. 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automationcited by 70
  5. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 64
  6. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 63
  7. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  8. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  9. 2024AdvAgent: Controllable Blackbox Red-teaming on Web Agentscited by 57
  10. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflowscited by 57
  11. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 55
  12. 2024ImgTrojan: Jailbreaking Vision-Language Models with ONE Imagecited by 50
  13. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  14. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  15. 2025VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agentscited by 42
  16. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injectioncited by 39
  17. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  18. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  19. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
  20. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  21. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  22. 2025Multi-Agent Security Tax: Trading Off Security and Collaboration Capabilities in Multi-Agent Systemscited by 27
  23. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  24. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systemscited by 24
  25. 2024Can We Trust Embodied Agents? Exploring Backdoor Attacks against Embodied LLM-based Decision-Making Systemscited by 23
  26. 2025DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM-based Agentcited by 23
  27. 2025Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agentscited by 23
  28. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  29. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 22
  30. 2024BadActs: A Universal Backdoor Defense in the Activation Spacecited by 19
  31. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 17
  32. 2025LLM DNA: Tracing Model Evolution via Functional Representationscited by 15
  33. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 14
  34. 2026Securing LLM-Agent Long-Term Memory Against Poisoning: Non-Malleable, Origin-Bound Authority with Machine-Checked Guaranteescited by 14
  35. 2024Future Events as Backdoor Triggers: Investigating Temporal Vulnerabilities in LLMscited by 13
  36. 2025Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agentscited by 12
  37. 2026BackdoorAgent: A Unified Framework for Backdoor Attacks on LLM-based Agentscited by 12
  38. 2024Exploring Backdoor Vulnerabilities of Chat Modelscited by 11

50 most cited of 135 All 135 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents
  2. 2025A Survey of Attacks on Large Language Models
  3. 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Models
  4. 2026Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Use
  5. 2025Rethinking Reasoning: A Survey on Reasoning-based Backdoors in LLMs
  6. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation
  7. 2025Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents
  8. 2024Imprompter: Tricking LLM Agents into Improper Tool Use