mlsanityData to 7 Oct 2026

AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases

arXiv 2407.12784 ↗First posted 17 Jul 2024cs.LG cs.CR cs.IRcited by 402

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

LLM agents have demonstrated remarkable performance across various applications, primarily due to their advanced capabilities in reasoning, utilizing external knowledge and tools, calling APIs, and executing actions to interact with environments. Current agents typically utilize a memory module or a retrieval-augmented generation (RAG) mechanism, retrieving past knowledge and instances with similar embeddings from knowledge bases to inform task planning and execution. However, the reliance on unverified knowledge bases raises significant concerns about their safety and trustworthiness. To uncover such vulnerabilities, we propose a novel red teaming approach AgentPoison, the first backdoor attack targeting generic and RAG-based LLM agents by poisoning their long-term memory or RAG knowledge base. In particular, we form the trigger generation process as a constrained optimization to optimize backdoor triggers by mapping the triggered instances to a unique embedding space, so as to ensure that whenever a user instruction contains the optimized backdoor trigger, the malicious demonstrations are retrieved from the poisoned memory or knowledge base with high probability. In the meantime, benign instructions without the trigger will still maintain normal performance. Unlike conventional backdoor attacks, AgentPoison requires no additional model training or fine-tuning, and the optimized backdoor trigger exhibits superior transferability, in-context coherence, and stealthiness. Extensive experiments demonstrate AgentPoison's effectiveness in attacking three types of real-world LLM agents: RAG-based autonomous driving agent, knowledge-intensive QA agent, and healthcare EHRAgent. On each agent, AgentPoison achieves an average attack success rate higher than 80% with minimal impact on benign performance (less than 1%) with a poison rate less than 0.1%.

Key facts. Cited by 402 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 29 of its 42 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

29 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents17 Feb

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  7. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  8. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  9. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  10. 2022PAL: Program-aided Language Modelscited by 635
  11. 2023Detecting Language Model Attacks with Perplexitycited by 411
  12. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
16 more
  1. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  2. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  3. 2023A Language Agent for Autonomous Drivingcited by 161
  4. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 153
  5. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  6. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 144
  7. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  8. 2023Backdoor Attacks for In-Context Learning with Language Modelscited by 92
  9. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 91
  10. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  11. 2024EHRAgent: Code Empowers Large Language Models for Few-shot Complex Tabular Reasoning on Electronic Health Recordscited by 75
  12. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 74
  13. 2023A Multi-Task Embedder For Retrieval Augmented LLMscited by 63
  14. 2024RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Modelscited by 58
  15. 2023Personalized Autonomous Driving with Large Language Models: Field Experimentscited by 46
  16. 2023PoisonPrompt: Backdoor Attack on Prompt-based Large Language Modelscited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 127; 2026: 261.

402 papers.

Sort:

By area:

  1. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  2. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  3. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
  4. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  5. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  6. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  7. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoningcited by 144
  8. 2025Memory Injection Attacks on LLM Agents via Query-Only Interactioncited by 127
  9. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  10. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 98
  11. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  12. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
38 more
  1. 2026Skill-Inject: Measuring Agent Vulnerability to Skill File Attackscited by 82
  2. 2025MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrievalcited by 65
  3. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  4. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  5. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  6. 2025A-MemGuard: A Proactive Defense Framework for LLM-Based Agent Memorycited by 61
  7. 2025G-Safeguard: A Topology-Guided Security Lens and Treatment on LLM-based Multi-agent Systemscited by 56
  8. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  9. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 50
  10. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  11. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  12. 2025AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agentscited by 42
  13. 2026Memory Poisoning Attack and Defense on Memory Based LLM-Agentscited by 40
  14. 2024SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanationscited by 37
  15. 2025BlindGuard: Safeguarding LLM-based Multi-Agent Systems under Unknown Attackscited by 37
  16. 2025AgentSafe: Safeguarding Large Language Model-based Multi-agent Systems via Hierarchical Data Managementcited by 35
  17. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 33
  18. 2025Securing Agentic AI: A Comprehensive Threat Model and Mitigation Framework for Generative AI Agentscited by 32
  19. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 32
  20. 2025DoomArena: A framework for Testing AI Agents Against Evolving Security Threatscited by 31
  21. 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agentscited by 31
  22. 2026AgentLAB: Benchmarking LLM Agents against Long-Horizon Attackscited by 31
  23. 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agentscited by 29
  24. 2025LLM Agents Should Employ Security Principlescited by 28
  25. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  26. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  27. 2026From Assistant to Double Agent: Formalizing and Benchmarking Attacks on OpenClaw for Personalized Local AI Agentcited by 27
  28. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  29. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  30. 2025Large Language Models Miss the Multi-Agent Markcited by 25
  31. 2026BadSkill: Backdoor Attacks on Agent Skills via Model-in-Skill Poisoningcited by 25
  32. 2025GraphRAG under Firecited by 23
  33. 2025Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agentscited by 23
  34. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  35. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  36. 2026From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agentscited by 23
  37. 2025ToolTweak: An Attack on Tool Selection in LLM-based Agentscited by 22
  38. 2024Backdoored Retrievers for Prompt Injection Attacks on Retrieval Augmented Generation of Large Language Modelscited by 21

50 most cited of 402 All 402 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Stop Fixating on Prompts: Reasoning Hijacking and Constraint Tightening for Red-Teaming LLM Agents
  2. 2025UDora: A Unified Red Teaming Framework against LLM Agents by Dynamically Hijacking Their Own Reasoning
  3. 2026REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems
  4. 2025SIRAJ: Diverse and Efficient Red-Teaming for LLM Agents via Distilled Structured Reasoning
  5. 2026Agent Against Agent: An Agentic System for Automatic Prompt Injection Red Teaming
  6. 2026RedEvoAgent: Automatic Red-Teaming Agent with Experience-Driven Skill Evolution
  7. 2026PI-Hunter: Automated Red-Teaming for Exposing and Localizing Prompt Injections
  8. 2026AgenticRed: Evolving Agentic Systems for Red-Teaming