# Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreaking

arXiv: https://arxiv.org/abs/2409.08045 · HTML: https://mlsanity.com/paper/2409.08045/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Membership inference attacks, one of 12 directions in LLM safety and security.
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 12 Sep 2024.

## Placed in

- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 26 of 26)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- ["Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://mlsanity.com/paper/2308.03825/) (2023)
- [ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledge](https://mlsanity.com/paper/2303.14070/) (2023)
- [Low-Resource Languages Jailbreak GPT-4](https://mlsanity.com/paper/2310.02446/) (2023)
- [Nomic Embed: Training a Reproducible Long Context Text Embedder](https://mlsanity.com/paper/2402.01613/) (2024)
- [SneakyPrompt: Jailbreaking Text-to-image Generative Models](https://mlsanity.com/paper/2305.12082/) (2023)
- [The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)](https://mlsanity.com/paper/2402.16893/) (2024)
- [PLeak: Prompt Leaking Attacks against Large Language Model Applications](https://mlsanity.com/paper/2405.06823/) (2024)
- [Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documents](https://mlsanity.com/paper/2406.05870/) (2024)
- [Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models](https://mlsanity.com/paper/2403.17336/) (2024)
- [BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models](https://mlsanity.com/paper/2406.00083/) (2024)
- [Gecko: Versatile Text Embeddings Distilled from Large Language Models](https://mlsanity.com/paper/2403.20327/) (2024)
- [Language Model Inversion](https://mlsanity.com/paper/2311.13647/) (2023)
- [TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Models](https://mlsanity.com/paper/2405.13401/) (2024)
- [Query-Based Adversarial Prompt Generation](https://mlsanity.com/paper/2402.12329/) (2024)
- [Phantom: General Backdoor Attacks on Retrieval Augmented Language Generation](https://mlsanity.com/paper/2405.20485/) (2024)
- [Is My Data in Your Retrieval Database? Membership Inference Attacks Against Retrieval Augmented Generation](https://mlsanity.com/paper/2405.20446/) (2024)
- [Prompt Stealing Attacks Against Large Language Models](https://mlsanity.com/paper/2402.12959/) (2024)
- [Automated Behavioral Analysis of Malware A Case Study of WannaCry Ransomware](https://mlsanity.com/paper/1709.08753/) (2017)

## Built on by (most cited first, 30 of 30)

- [Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applications](https://mlsanity.com/paper/2411.14110/) (2024)
- [Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2502.06872/) (2025)
- [Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.00306/) (2025)
- [Silent Leaks: Implicit Knowledge Extraction Attack on RAG Systems through Benign Queries](https://mlsanity.com/paper/2505.15420/) (2025)
- [Pirates of the RAG: Adaptively Attacking LLMs to Leak Knowledge Bases](https://mlsanity.com/paper/2412.18295/) (2024)
- [Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models](https://mlsanity.com/paper/2503.01742/) (2025)
- [Privacy-Aware RAG: Secure and Isolated Knowledge Retrieval](https://mlsanity.com/paper/2503.15548/) (2025)
- [SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2601.03979/) (2026)
- [Exposing Privacy Risks in Graph Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.17222/) (2025)
- [CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMs](https://mlsanity.com/paper/2510.09871/) (2025)
- [ADAM: A Systematic Data Extraction Attack on Agent Memory via Adaptive Querying](https://mlsanity.com/paper/2604.09747/) (2026)
- [Towards Copyright Protection for Knowledge Bases of Retrieval-augmented Language Models via Reasoning](https://mlsanity.com/paper/2502.10440/) (2025)
- [RAG-Pull: Turning Retrieval into a Code-Injection Channel via Invisible Unicode Perturbations](https://mlsanity.com/paper/2510.11195/) (2025)
- [Five Queries Are Enough: Query-Efficient and Surrogate-Free Membership Inference Attacks on RAG via Entailment](https://mlsanity.com/paper/2605.24312/) (2026)
- [Query-Efficient Agentic Graph Extraction Attacks on GraphRAG Systems](https://mlsanity.com/paper/2601.14662/) (2026)
- [Connect the Dots: Knowledge Graph-Guided Crawler Attack on Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2601.15678/) (2026)
- [External Data Extraction Attacks against Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2510.02964/) (2025)
- [MemPot: Defending Against Memory Extraction Attack with Optimized Honeypots](https://mlsanity.com/paper/2602.07517/) (2026)
- [Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generation](https://mlsanity.com/paper/2508.10404/) (2025)
- [MES-RAG: Bringing Multi-modal, Entity-Storage, and Secure Enhancements to RAG](https://mlsanity.com/paper/2503.13563/) (2025)
- [Provably Secure Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.01084/) (2025)
- [RAGFort: Dual-Path Defense Against Proprietary Knowledge Base Extraction in Retrieval-Augmented Generation](https://mlsanity.com/paper/2511.10128/) (2025)
- [Graphs Don't Stay Secret: Practical Subgraph Reconstruction Attacks on Defended Graph RAG](https://mlsanity.com/paper/2602.06495/) (2026)
- [Detecting RAG Extraction Attack via Dual-Path Runtime Integrity Game](https://mlsanity.com/paper/2604.10717/) (2026)
- [A Case Study on the Impact of Anonymization Along the RAG Pipeline](https://mlsanity.com/paper/2604.15958/) (2026)
- [LeakDojo: Decoding the Leakage Threats of RAG Systems](https://mlsanity.com/paper/2605.05818/) (2026)
- [ALDEN: Boosting Private Data Extraction from Retrieval-Augmented Generation Systems via Active Learning and Distribution Estimation](https://mlsanity.com/paper/2605.18762/) (2026)
- [Extracting Knowledge from Tools in LLM Agents](https://mlsanity.com/paper/2608.30288/) (2026)
- [RAG-CT: Mitigating Privacy Risks on Retrieval-Augmented Generation Systems via Scanning Prompt Distribution](https://mlsanity.com/paper/2609.16095/) (2026)
- [Walking the Embedding Space: Datastore Extraction from Multimodal RAG](https://mlsanity.com/paper/2610.01871/) (2026)

## Abstract (arXiv)

In this paper, we show that with the ability to jailbreak a GenAI model, attackers can escalate the outcome of attacks against RAG-based GenAI-powered applications in severity and scale. In the first part of the paper, we show that attackers can escalate RAG membership inference attacks and RAG entity extraction attacks to RAG documents extraction attacks, forcing a more severe outcome compared to existing attacks. We evaluate the results obtained from three extraction methods, the influence of the type and the size of five embeddings algorithms employed, the size of the provided context, and the GenAI engine. We show that attackers can extract 80%-99.8% of the data stored in the database used by the RAG of a Q&A chatbot. In the second part of the paper, we show that attackers can escalate the scale of RAG data poisoning attacks from compromising a single GenAI-powered application to compromising the entire GenAI ecosystem, forcing a greater scale of damage. This is done by crafting an adversarial self-replicating prompt that triggers a chain reaction of a computer worm within the ecosystem and forces each affected application to perform a malicious activity and compromise the RAG of additional applications. We evaluate the performance of the worm in creating a chain of confidential data extraction about users within a GenAI ecosystem of GenAI-powered email assistants and analyze how the performance of the worm is affected by the size of the context, the adversarial self-replicating prompt used, the type and size of the embeddings algorithm employed, and the number of hops in the propagation. Finally, we review and analyze guardrails to protect RAG-based inference and discuss the tradeoffs.