mlsanityData to 7 Oct 2026

BadAgent: Inserting and Activating Backdoor Attacks in LLM Agents

arXiv 2406.03007 ↗First posted 5 Jun 2024cs.CL cs.AI cs.CRcited by 106

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

With the prosperity of large language models (LLMs), powerful LLM-based intelligent agents have been developed to provide customized services with a set of user-defined tools. State-of-the-art methods for constructing LLM agents adopt trained LLMs and further fine-tune them on data for the agent task. However, we show that such methods are vulnerable to our proposed backdoor attacks named BadAgent on various agent tasks, where a backdoor can be embedded by fine-tuning on the backdoor data. At test time, the attacker can manipulate the deployed LLM agents to execute harmful operations by showing the trigger in the agent input or environment. To our surprise, our proposed attack methods are extremely robust even after fine-tuning on trustworthy data. Though backdoor attacks have been studied extensively in natural language processing, to the best of our knowledge, we could be the first to study them on LLM agents that are more dangerous due to the permission to use external tools. Our work demonstrates the clear risk of constructing LLM agents based on untrusted LLMs or data. Our code is public at https://github.com/DPamK/BadAgent

Key facts. Cited by 106 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 33 of its 51 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

33 of 51 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  11. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  12. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
21 more
  1. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  2. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  3. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
  4. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  5. 2023Language Models can Solve Computer Taskscited by 371
  6. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  7. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  8. 2023Poisoning Language Models During Instruction Tuningcited by 254
  9. 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 250
  10. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 203
  11. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  12. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 174
  13. 2020Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Reviewcited by 120
  14. 2021Hidden Backdoors in Human-Centric Language Modelscited by 83
  15. 2021BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Modelscited by 76
  16. 2023BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPTcited by 70
  17. 2023From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?cited by 53
  18. 2023Backdoor Attacks and Countermeasures in Natural Language Processing Models: A Comprehensive Security Reviewcited by 34
  19. 2023Last One Standing: A Comparative Analysis of Security and Privacy of Soft Prompt Tuning, LoRA, and In-Context Learningcited by 13
  20. 2023A Survey on Interpretable Cross-modal Reasoningcited by 2
  21. 2023Erasing Self-Supervised Learning Backdoor by Cluster Activation Maskingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 47; 2026: 51.

106 papers.

Sort:

By area:

  1. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  2. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  3. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  4. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  6. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  7. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 64
  8. 2024AdvAgent: Controllable Blackbox Red-teaming on Web Agentscited by 57
  9. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflowscited by 57
  10. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  11. 2025VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agentscited by 42
  12. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injectioncited by 39
38 more
  1. 2024Denial-of-Service Poisoning Attacks against Large Language Modelscited by 34
  2. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
  3. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  4. 2025Adversarial Attacks on Robotic Vision Language Action Modelscited by 31
  5. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  6. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  7. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  8. 2025TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systemscited by 24
  9. 2025DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM-based Agentcited by 23
  10. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  11. 2026ClawKeeper: Comprehensive Safety Protection for OpenClaw Agents Through Skills, Plugins, and Watcherscited by 18
  12. 2026Poison Once, Exploit Forever: Environment-Injected Memory Poisoning Attacks on Web Agentscited by 18
  13. 2026Intelligent AI Delegationcited by 16
  14. 2026AgentWorm: Self-Propagating Attacks Across LLM Agent Ecosystemscited by 15
  15. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 14
  16. 2025A Survey of Attacks on Large Language Modelscited by 13
  17. 2025UniGuardian: A Unified Defense for Detecting Prompt Injection, Backdoor Attacks and Adversarial Attacks in Large Language Modelscited by 12
  18. 2025Hidden Ghost Hand: Unveiling Backdoor Vulnerabilities in MLLM-Powered Mobile GUI Agentscited by 12
  19. 2025Who's the Mole? Modeling and Detecting Intention-Hiding Malicious Agents in LLM-Based Multi-Agent Systemscited by 12
  20. 2026BackdoorAgent: A Unified Framework for Backdoor Attacks on LLM-based Agentscited by 12
  21. 2025Are Your Agents Upward Deceivers?cited by 11
  22. 2026Sponge Tool Attack: Stealthy Denial-of-Efficiency against Tool-Augmented Agentic Reasoningcited by 11
  23. 2025Mimicking the Familiar: Dynamic Command Generation for Information Theft Attacks in LLM Tool-Learning Systemcited by 10
  24. 2025Cloud Infrastructure Management in the Age of AI Agentscited by 9
  25. 2026Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Usecited by 9
  26. 2026HarmfulSkillBench: How Do Harmful Skills Weaponize Your Agents?cited by 9
  27. 2025A Systematic Review of Poisoning Attacks Against Large Language Modelscited by 8
  28. 2025Web Fraud Attacks Against LLM-Driven Multi-Agent Systemscited by 8
  29. 2025Backdoor Attribution: Elucidating and Controlling Backdoor in Language Modelscited by 8
  30. 2026Risky-Bench: Probing Agentic Safety Risks under Real-World Deploymentcited by 8
  31. 2026LPS-Bench: Benchmarking Safety Awareness of Computer-Use Agents in Long-Horizon Planning under Benign and Adversarial Scenarioscited by 8
  32. 2026SOK: A Taxonomy of Attack Vectors and Defense Strategies for Agentic Supply Chain Runtimecited by 8
  33. 2025TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agentcited by 7
  34. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 7
  35. 2025AutoBackdoor: Automating Backdoor Attacks via LLM Agentscited by 7
  36. 2025Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chaincited by 6
  37. 2026Behavioral Integrity Verification for AI Agent Skillscited by 6
  38. 2024Systematic Categorization, Construction and Evaluation of New Attacks against Multi-modal Mobile GUI Agentscited by 5

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents
  2. 2024Imprompter: Tricking LLM Agents into Improper Tool Use
  3. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
  4. 2025CheatAgent: Attacking LLM-Empowered Recommender Systems via LLM Agent
  5. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents
  6. 2025Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents
  7. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases
  8. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks