mlsanityData to 14 Oct 2026

"Glue pizza and eat rocks" -- Exploiting Vulnerabilities in Retrieval-Augmented Generative Models

arXiv 2406.19417 ↗First posted 26 Jun 2024cs.CR cs.AIcited by 60

On the map: Retrieval-augmented generation → Privacy and security of RAG

Abstract (arXiv)

Retrieval-Augmented Generative (RAG) models enhance Large Language Models (LLMs) by integrating external knowledge bases, improving their performance in applications like fact-checking and information searching. In this paper, we demonstrate a security threat where adversaries can exploit the openness of these knowledge bases by injecting deceptive content into the retrieval database, intentionally changing the model's behavior. This threat is critical as it mirrors real-world usage scenarios where RAG systems interact with publicly accessible knowledge bases, such as web scrapings and user-contributed data pools. To be more realistic, we target a realistic setting where the adversary has no knowledge of users' queries, knowledge base data, and the LLM parameters. We demonstrate that it is possible to exploit the model successfully through crafted content uploads with access to the retriever. Our findings emphasize an urgent need for security measures in the design and deployment of RAG systems to prevent potential manipulation and ensure the integrity of machine-generated content.

Key facts. Cited by 60 papers in the map (data through 14 Oct 2026). Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation. Builds on 31 of its 64 references that are in the map. First posted on arXiv 26 Jun 2024.

Builds on

31 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations22 Apr

Other references · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  12. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
18 more
  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  3. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  5. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 498
  6. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  7. 2021Internet-Augmented Dialogue Generationcited by 217
  8. 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 159
  9. 2022A Survey on Retrieval-Augmented Text Generationcited by 158
  10. 2021Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyondcited by 153
  11. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89
  12. 2022A Study of the Attention Abnormality in Trojaned BERTscited by 48
  13. 2020Adversarial Semantic Collisionscited by 42
  14. 2023Black-box Adversarial Attacks against Dense Retrieval Models: A Multi-view Contrastive Learning Methodcited by 29
  15. 2020One word at a time: adversarial attacks on retrieval modelscited by 23
  16. 2024Task-Agnostic Detector for Insertion-Based Backdoor Attackscited by 23
  17. 2024The Wolf Within: Covert Injection of Malice into MLLM Societies via an MLLM Operativecited by 21
  18. 2024MARIO Eval: Evaluate Your Math LLM with your Math LLM--A mathematical dataset evaluation toolkitcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 34; 2026: 20.

60 papers.

Sort:

By area:

  1. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  2. 2025TrustRAG: Enhancing Robustness and Trustworthiness in Retrieval-Augmented Generationcited by 40
  3. 2024SMoA: Improving Multi-agent Large Language Models with Sparse Mixture-of-Agentscited by 30
  4. 2025Traceback of Poisoning Attacks to Retrieval-Augmented Generationcited by 28
  5. 2025One Shot Dominance: Knowledge Poisoning Attack on Retrieval-Augmented Generation Systemscited by 28
  6. 2025AnyMAC: Cascading Flexible Multi-Agent Collaboration via Next-Agent Predictioncited by 25
  7. 2025GraphRAG under Firecited by 24
  8. 2025MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attackscited by 24
  9. 2024Towards Resilient and Efficient LLMs: A Comparative Study of Efficiency, Performance, and Adversarial Robustnesscited by 22
  10. 2025Benchmarking Poisoning Attacks against Retrieval-Augmented Generationcited by 18
  11. 2025CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Modelscited by 17
  12. 2025Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Searchcited by 16
38 more
  1. 2025Joint-GCG: Unified Gradient-Based Poisoning Attacks on Retrieval-Augmented Generation Systemscited by 16
  2. 2024Towards More Robust Retrieval-Augmented Generation: Evaluating RAG Under Adversarial Poisoning Attackscited by 14
  3. 2025The RAG Paradox: A Black-Box Attack Exploiting Unintentional Vulnerabilities in Retrieval-Augmented Generation Systemscited by 14
  4. 2025Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generationcited by 12
  5. 2025Who Taught the Lie? Responsibility Attribution for Poisoned Knowledge in Retrieval-Augmented Generationcited by 12
  6. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  7. 2024Ontology-Aware RAG for Improved Question-Answering in Cybersecurity Educationcited by 9
  8. 2025One Pic is All it Takes: Poisoning Visual Document Retrieval Augmented Generation with a Single Imagecited by 8
  9. 2025RAG Security and Privacy: Formalizing the Threat Model and Attack Surfacecited by 8
  10. 2026SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systemscited by 7
  11. 2025LLMs as World Models: Data-Driven and Human-Centered Pre-Event Simulation for Disaster Impact Assessmentcited by 6
  12. 2025CtrlRAG: Black-box Document Poisoning Attacks for Retrieval-Augmented Generation of Large Language Modelscited by 5
  13. 2025Uncovering Competing Poisoning Attacks in Retrieval-Augmented Generationcited by 5
  14. 2025Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detectioncited by 5
  15. 2025AIP: Subverting Retrieval-Augmented Generation via Adversarial Instructional Promptcited by 5
  16. 2025RIPRAG: Hack a Black-box Retrieval-Augmented Generation Question-Answering System with Reinforcement Learningcited by 5
  17. 2025EcoSafeRAG: Efficient Security through Context Analysis in Retrieval-Augmented Generationcited by 4
  18. 2025SafeSearch: Automated Red-Teaming of LLM-Based Search Agentscited by 4
  19. 2025MIRAGE: Misleading Retrieval-Augmented Generation via Black-box and Query-agnostic Poisoning Attackscited by 4
  20. 2025Hoist with His Own Petard: Inducing Guardrails to Facilitate Denial-of-Service Attacks on Retrieval-Augmented Generation of LLMscited by 3
  21. 2025RAG Safety: Exploring Knowledge Poisoning Attacks to Retrieval-Augmented Generationcited by 3
  22. 2025UniC-RAG: Universal Knowledge Corruption Attacks to Retrieval-Augmented Generationcited by 3
  23. 2024Model Attribution in LLM-Generated Disinformation: A Domain Generalization Approach with Supervised Contrastive Learningcited by 2
  24. 2026ProGRank: Probe-Gradient Reranking to Defend Dense-Retriever RAG from Corpus Poisoningcited by 2
  25. 2026Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directionscited by 2
  26. 2025CyberBOT: Towards Reliable Cybersecurity Education via Ontology-Grounded Retrieval Augmented Generationcited by 1
  27. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  28. 2026KEPo: Knowledge Evolution Poison on Graph-based Retrieval-Augmented Generationcited by 1
  29. 2026Needle-in-RAG: Prompt-Conditioned Character-Level Traceback of Poisoned Spans in Retrieved Evidencecited by 1
  30. 2026DiscourseFlip: An Oblique Discourse-Level Opinion Manipulation Attack against Black-box Retrieval-Augmented Generationcited by 1
  31. 2026PRA-RAG: Provably Robust Aggregation in Retrieval-Augmented Generation against Retrieval Corruptioncited by 1
  32. 2026Retrieved But Not Reliable: A Survey on Attacks, and Defenses in Retrieval-Augmented Generationcited by 1
  33. 2025The Other Side of the Coin: Exploring Fairness in Retrieval-Augmented Generationcited by 0
  34. 2025When GPT Spills the Tea: Comprehensive Assessment of Knowledge File Leakage in GPTscited by 0
  35. 2025Provable one-poison backdoor attacks on linear models and ReLU neural networkscited by 0
  36. 2025Separate the Wheat from the Chaff: Winnowing Down Divergent Views in Retrieval Augmented Generationcited by 0
  37. 2026Hidden in the Metadata: Stealth Poisoning Attacks on Multimodal Retrieval-Augmented Generationcited by 0
  38. 2026Knowledge Poisoning Attacks on Medical Multi-Modal Retrieval-Augmented Generationcited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systems
  2. 2024Pirates of the RAG: Adaptively Attacking LLMs to Leak Knowledge Bases
  3. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models
  4. 2024BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models
  5. 2026SD-RAG: A Prompt-Injection-Resilient Framework for Selective Disclosure in Retrieval-Augmented Generation
  6. 2025Token-Level Precise Attack on RAG: Searching for the Best Alternatives to Mislead Generation
  7. 2024Data Extraction Attacks in Retrieval-Augmented Generation via Backdoors
  8. 2026Mitigating Database Leakage in RAG Systems with Keyword-Grounded Fact Substitution