mlsanityData to 7 Oct 2026

PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models

arXiv 2402.07867 ↗First posted 12 Feb 2024cs.CR cs.LGcited by 251

On the map: Retrieval-augmented generation → Privacy and security of RAG

Abstract (arXiv)

Large language models (LLMs) have achieved remarkable success due to their exceptional generative capabilities. Despite their success, they also have inherent limitations such as a lack of up-to-date knowledge and hallucination. Retrieval-Augmented Generation (RAG) is a state-of-the-art technique to mitigate these limitations. The key idea of RAG is to ground the answer generation of an LLM on external knowledge retrieved from a knowledge database. Existing studies mainly focus on improving the accuracy or efficiency of RAG, leaving its security largely unexplored. We aim to bridge the gap in this work. We find that the knowledge database in a RAG system introduces a new and practical attack surface. Based on this attack surface, we propose PoisonedRAG, the first knowledge corruption attack to RAG, where an attacker could inject a few malicious texts into the knowledge database of a RAG system to induce an LLM to generate an attacker-chosen target answer for an attacker-chosen target question. We formulate knowledge corruption attacks as an optimization problem, whose solution is a set of malicious texts. Depending on the background knowledge (e.g., black-box and white-box settings) of an attacker on a RAG system, we propose two solutions to solve the optimization problem, respectively. Our results show PoisonedRAG could achieve a 90% attack success rate when injecting five malicious texts for each target question into a knowledge database with millions of texts. We also evaluate several defenses and our results show they are insufficient to defend against PoisonedRAG, highlighting the need for new defenses.

Key facts. Cited by 251 papers in the map (data through 7 Oct 2026). Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation. Builds on 69 of its 105 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

69 of 105 references.

References · 69

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
38 more
  1. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  4. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  5. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  6. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  7. 2023PaLM 2 Technical Reportcited by 1,154
  8. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  9. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  10. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  11. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  12. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  13. 2018How To Backdoor Federated Learningcited by 745
  14. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  15. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  16. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  17. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  18. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  19. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 479
  20. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  21. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  22. 2023Detecting Language Model Attacks with Perplexitycited by 411
  23. 2018TextBugger: Generating Adversarial Text Against Real-world Applicationscited by 376
  24. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  25. 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 341
  26. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  27. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  28. 2017Certified Defenses for Data Poisoning Attackscited by 288
  29. 2023Poisoning Web-Scale Training Datasets is Practicalcited by 276
  30. 2023Poisoning Language Models During Instruction Tuningcited by 254
  31. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  32. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  33. 2023MasterKey: Automated Jailbreak Across Multiple Large Language Model Chatbotscited by 172
  34. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 153
  35. 2021Poisoning and Backdooring Contrastive Learningcited by 149
  36. 2020Backdoor Attacks to Graph Neural Networkscited by 126
  37. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 124
  38. 2023AnglE-optimized Text Embeddingscited by 98

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 74; 2026: 170.

251 papers.

Sort:

By area:

  1. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  2. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 144
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  5. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  6. 2025MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrievalcited by 65
  7. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  8. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  9. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 61
  10. 2025Retrieval-Augmented Generation with Conflicting Evidencecited by 60
  11. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflowscited by 57
  12. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
38 more
  1. 2026Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injectionscited by 44
  2. 2026Supply-Chain Poisoning Attacks Against LLM Coding Agent Skill Ecosystemscited by 42
  3. 2026Memory Poisoning Attack and Defense on Memory Based LLM-Agentscited by 40
  4. 2025Poison-RAG: Adversarial Data Poisoning Attacks on Retrieval-Augmented Generation in Recommender Systemscited by 32
  5. 2025Topic-FlipRAG: Topic-Orientated Adversarial Opinion Manipulation Attacks to Retrieval-Augmented Generation Modelscited by 32
  6. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  7. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  8. 2025GraphRAG under Firecited by 23
  9. 2025Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agentscited by 23
  10. 2025ToolTweak: An Attack on Tool Selection in LLM-based Agentscited by 22
  11. 2025A Comprehensive Survey on Reinforcement Learning-based Agentic Search: Foundations, Roles, Optimizations, Evaluations, and Applicationscited by 21
  12. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  13. 2025Secure Retrieval-Augmented Generation against Poisoning Attackscited by 19
  14. 2026Your Agent, Their Asset: A Real-World Safety Analysis of OpenClawcited by 19
  15. 2026Poison Once, Exploit Forever: Environment-Injected Memory Poisoning Attacks on Web Agentscited by 18
  16. 2025ReliabilityRAG: Effective and Provably Robust Defense for RAG-based Web-Searchcited by 17
  17. 2025ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Datacited by 17
  18. 2026MalTool: Malicious Tool Attacks on LLM Agentscited by 17
  19. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  20. 2025Position: Privacy Is Not Just Memorization!cited by 15
  21. 2026Does RAG Know When Retrieval Is Wrong? Diagnosing Context Compliance under Knowledge Conflictcited by 15
  22. 2025System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injectioncited by 14
  23. 2026Security Threat Modeling for Emerging AI-Agent Protocols: A Comparative Analysis of MCP, A2A, Agora, and ANPcited by 14
  24. 2026Securing LLM-Agent Long-Term Memory Against Poisoning: Non-Malleable, Origin-Bound Authority with Machine-Checked Guaranteescited by 14
  25. 2025A Survey of Attacks on Large Language Modelscited by 13
  26. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 13
  27. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injectioncited by 13
  28. 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generationcited by 12
  29. 2025SeCon-RAG: A Two-Stage Semantic Filtering and Conflict-Free Framework for Trustworthy RAGcited by 12
  30. 2026BackdoorAgent: A Unified Framework for Backdoor Attacks on LLM-based Agentscited by 12
  31. 2025TracLLM: A Generic Framework for Attributing Long Context LLMscited by 11
  32. 2025Defending Against Knowledge Poisoning Attacks During Retrieval-Augmented Generationcited by 11
  33. 2025ADMIT: Few-shot Knowledge Poisoning Attacks on RAG-based Fact Checkingcited by 9
  34. 2025Systems Security Foundations for Agentic Computingcited by 9
  35. 2026A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Frameworkcited by 9
  36. 2025Universal Zero-shot Embedding Inversioncited by 8
  37. 2025One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Imagecited by 8
  38. 2025Bias Amplification in RAG: Poisoning Knowledge Retrieval to Steer LLMscited by 8

50 most cited of 251 All 251 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Practical Poisoning Attacks against Retrieval-Augmented Generation
  2. 2025One Shot Dominance: Knowledge Poisoning Attack on Retrieval-Augmented Generation Systems
  3. 2024Data Extraction Attacks in Retrieval-Augmented Generation via Backdoors
  4. 2025Rescuing the Unpoisoned: Efficient Defense against Knowledge Corruption Attacks on RAG Systems
  5. 2025Token-Level Precise Attack on RAG: Searching for the Best Alternatives to Mislead Generation
  6. 2025RAG Safety: Exploring Knowledge Poisoning Attacks to Retrieval-Augmented Generation
  7. 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimization
  8. 2025Traceback of Poisoning Attacks to Retrieval-Augmented Generation