mlsanityData to 14 Oct 2026

Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreaking

arXiv 2409.08045 ↗First posted 12 Sep 2024cs.CR cs.AIcited by 30

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

In this paper, we show that with the ability to jailbreak a GenAI model, attackers can escalate the outcome of attacks against RAG-based GenAI-powered applications in severity and scale. In the first part of the paper, we show that attackers can escalate RAG membership inference attacks and RAG entity extraction attacks to RAG documents extraction attacks, forcing a more severe outcome compared to existing attacks. We evaluate the results obtained from three extraction methods, the influence of the type and the size of five embeddings algorithms employed, the size of the provided context, and the GenAI engine. We show that attackers can extract 80%-99.8% of the data stored in the database used by the RAG of a Q&A chatbot. In the second part of the paper, we show that attackers can escalate the scale of RAG data poisoning attacks from compromising a single GenAI-powered application to compromising the entire GenAI ecosystem, forcing a greater scale of damage. This is done by crafting an adversarial self-replicating prompt that triggers a chain reaction of a computer worm within the ecosystem and forces each affected application to perform a malicious activity and compromise the RAG of additional applications. We evaluate the performance of the worm in creating a chain of confidential data extraction about users within a GenAI ecosystem of GenAI-powered email assistants and analyze how the performance of the worm is affected by the size of the context, the adversarial self-replicating prompt used, the type and size of the embeddings algorithm employed, and the number of hops in the propagation. Finally, we review and analyze guardrails to protect RAG-based inference and discuss the tradeoffs.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 26 of its 52 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

26 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Phantom: General Backdoor Attacks on Retrieval Augmented Language Generation30 May
  2. 2024Is My Data in Your Retrieval Database? Membership Inference Attacks Against Retrieval Augmented Generation30 May

Other references · 24

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  4. 2020Extracting Training Data from Large Language Modelscited by 2,189
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  6. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 981
  7. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  8. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  9. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  10. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  11. 2023Low-Resource Languages Jailbreak GPT-4cited by 309
  12. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 265
12 more
  1. 2023SneakyPrompt: Jailbreaking Text-to-image Generative Modelscited by 189
  2. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  3. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 127
  4. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 102
  5. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 86
  6. 2024BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Modelscited by 86
  7. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 83
  8. 2023Language Model Inversioncited by 68
  9. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 67
  10. 2024Query-Based Adversarial Prompt Generationcited by 62
  11. 2024Prompt Stealing Attacks Against Large Language Modelscited by 40
  12. 2017Automated Behavioral Analysis of Malware A Case Study of WannaCry Ransomwarecited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 14; 2026: 14.

30 papers.

Sort:

By area:

  1. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 37
  2. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 30
  3. 2025Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented Generationcited by 26
  4. 2025Silent Leaks: Implicit Knowledge Extraction Attack on RAG Systems through Benign Queriescited by 19
  5. 2024Pirates of the RAG: Adaptively Attacking LLMs to Leak Knowledge Basescited by 18
  6. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  7. 2025Privacy-Aware RAG: Secure and Isolated Knowledge Retrievalcited by 10
  8. 2026SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systemscited by 7
  9. 2025Exposing Privacy Risks in Graph Retrieval-Augmented Generationcited by 5
  10. 2025CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMscited by 5
  11. 2026ADAM: A Systematic Data Extraction Attack on Agent Memory via Adaptive Queryingcited by 5
  12. 2025Towards Copyright Protection for Knowledge Bases of Retrieval-augmented Language Models via Reasoningcited by 4
18 more
  1. 2025RAG-Pull: Turning Retrieval into a Code-Injection Channel via Invisible Unicode Perturbationscited by 4
  2. 2026Five Queries Are Enough: Query-Efficient and Surrogate-Free Membership Inference Attacks on RAG via Entailmentcited by 4
  3. 2026Query-Efficient Agentic Graph Extraction Attacks on GraphRAG Systemscited by 3
  4. 2026Connect the Dots: Knowledge Graph-Guided Crawler Attack on Retrieval-Augmented Generation Systemscited by 3
  5. 2025External Data Extraction Attacks against Retrieval-Augmented Large Language Modelscited by 2
  6. 2026MemPot: Defending Against Memory Extraction Attack with Optimized Honeypotscited by 2
  7. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  8. 2025MES-RAG: Bringing Multi-modal, Entity-Storage, and Secure Enhancements to RAGcited by 0
  9. 2025Provably Secure Retrieval-Augmented Generationcited by 0
  10. 2025RAGFort: Dual-Path Defense Against Proprietary Knowledge Base Extraction in Retrieval-Augmented Generationcited by 0
  11. 2026Graphs Don't Stay Secret: Practical Subgraph Reconstruction Attacks on Defended Graph RAGcited by 0
  12. 2026Detecting RAG Extraction Attack via Dual-Path Runtime Integrity Gamecited by 0
  13. 2026A Case Study on the Impact of Anonymization Along the RAG Pipelinecited by 0
  14. 2026LeakDojo: Decoding the Leakage Threats of RAG Systemscited by 0
  15. 2026ALDEN: Boosting Private Data Extraction from Retrieval-Augmented Generation Systems via Active Learning and Distribution Estimationcited by 0
  16. 2026Extracting Knowledge from Tools in LLM Agentscited by 0
  17. 2026RAG-CT: Mitigating Privacy Risks on Retrieval-Augmented Generation Systems via Scanning Prompt Distributioncited by 0
  18. 2026Walking the Embedding Space: Datastore Extraction from Multimodal RAGcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications
  2. 2024Data Extraction Attacks in Retrieval-Augmented Generation via Backdoors
  3. 2025RAG Safety: Exploring Knowledge Poisoning Attacks to Retrieval-Augmented Generation
  4. 2025Practical Poisoning Attacks against Retrieval-Augmented Generation
  5. 2025Rescuing the Unpoisoned: Efficient Defense against Knowledge Corruption Attacks on RAG Systems
  6. 2024Backdoored Retrievers for Prompt Injection Attacks on Retrieval Augmented Generation of Large Language Models
  7. 2025UniC-RAG: Universal Knowledge Corruption Attacks to Retrieval-Augmented Generation
  8. 2025RAGPart & RAGMask: Retrieval-Stage Defenses Against Corpus Poisoning in Retrieval-Augmented Generation