mlsanityData to 7 Oct 2026

Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems

arXiv 2402.17840 ↗First posted 27 Feb 2024cs.CL cs.AI cs.CRcited by 76

On the map: Retrieval-augmented generation → Privacy and security of RAG

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) improves pre-trained models by incorporating external knowledge at test time to enable customized adaptation. We study the risk of datastore leakage in Retrieval-In-Context RAG Language Models (LMs). We show that an adversary can exploit LMs' instruction-following capabilities to easily extract text data verbatim from the datastore of RAG systems built with instruction-tuned LMs via prompt injection. The vulnerability exists for a wide range of modern LMs that span Llama2, Mistral/Mixtral, Vicuna, SOLAR, WizardLM, Qwen1.5, and Platypus2, and the exploitability exacerbates as the model size scales up. We also study multiple effects of RAG setup on the extractability of data, indicating that following unexpected instructions to regurgitate data can be an outcome of failure in effectively utilizing contexts for modern LMs, and further show that such vulnerability can be greatly mitigated by position bias elimination strategies. Extending our study to production RAG models GPTs, we design an attack that can cause datastore leakage with a 100% success rate on 25 randomly selected customized GPTs with at most 2 queries, and we extract text data verbatim at a rate of 41% from a book of 77,000 words and 3% from a corpus of 1,569,000 words by prompting the GPTs with only 100 queries generated by themselves.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation. Builds on 55 of its 84 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

55 of 84 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)23 Feb

Other references · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  9. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
38 more
  1. 2024Mixtral of Expertscited by 1,785
  2. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  3. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  5. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  6. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  7. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  8. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  9. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  10. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  11. 2023Extracting Training Data from Diffusion Modelscited by 723
  12. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  13. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  14. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  15. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  16. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  17. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  18. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  19. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
  20. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
  21. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  22. 2022Teaching language models to support answers with verified quotescited by 290
  23. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  24. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  25. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  26. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  27. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  28. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  29. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  30. 2023What's In My Big Data?cited by 129
  31. 2022Retrieval-Augmented Multimodal Language Modelingcited by 125
  32. 2023Retrieval meets Long Context Large Language Modelscited by 114
  33. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  34. 2019Gmail Smart Compose: Real-Time Assisted Writingcited by 97
  35. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  36. 2023Attention Sorting Combats Recency Bias In Long Context Language Modelscited by 89
  37. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  38. 2023Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Modelscited by 67

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 23; 2026: 40.

76 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2025Unveiling Privacy Risks in LLM Agent Memorycited by 99
  3. 2024Trustworthiness in Retrieval-Augmented Generation Systems: A Surveycited by 83
  4. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  5. 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflowscited by 57
  6. 2024Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Datacited by 52
  7. 2024Privacy-Preserving Retrieval-Augmented Generation with Differential Privacycited by 38
  8. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 36
  9. 2024Mask-based Membership Inference Attacks for Retrieval-Augmented Generationcited by 29
  10. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 29
  11. 2024Prompt Leakage effect and defense strategies for multi-turn LLM interactionscited by 23
  12. 2024Data Extraction Attacks in Retrieval-Augmented Generation via Backdoorscited by 23
38 more
  1. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 22
  2. 2025Silent Leaks: Implicit Knowledge Extraction Attack on RAG Systems through Benign Queriescited by 19
  3. 2024RAG with Differential Privacycited by 18
  4. 2026Poison Once, Exploit Forever: Environment-Injected Memory Poisoning Attacks on Web Agentscited by 18
  5. 2024Pirates of the RAG: Adaptively Attacking LLMs to Leak Knowledge Basescited by 17
  6. 20251-2-3 Check: Enhancing Contextual Privacy in LLM via Multi-Agent Reasoningcited by 16
  7. 2026Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systemscited by 16
  8. 2025Dataset Protection via Watermarked Canaries in Retrieval-Augmented LLMscited by 11
  9. 2025Privacy-Aware RAG: Secure and Isolated Knowledge Retrievalcited by 10
  10. 2025PIG: Privacy Jailbreak Attack on LLMs via Gradient-based Iterative In-Context Optimizationcited by 10
  11. 2024Towards More Realistic Extraction Attacks: An Adversarial Perspectivecited by 8
  12. 2025LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasurescited by 8
  13. 2025Privacy-Aware Decoding: Mitigating Privacy Leakage of Large Language Models in Retrieval-Augmented Generationcited by 8
  14. 2025RAG Security and Privacy: Formalizing the Threat Model and Attack Surfacecited by 8
  15. 2025Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generationcited by 7
  16. 2026SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systemscited by 7
  17. 2024Estimating Privacy Leakage of Augmented Contextual Knowledge in Language Modelscited by 6
  18. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  19. 2025Differentially Private Synthetic Text Generation for Retrieval-Augmented Generation (RAG)cited by 5
  20. 2026ADAM: A Systematic Data Extraction Attack on Agent Memory via Adaptive Queryingcited by 5
  21. 2026Observable Channels, Not Just Storage: Evaluating Privacy Leakage in LLM Agent Pipelinescited by 4
  22. 2026Five Queries Are Enough: Query-Efficient and Surrogate-Free Membership Inference Attacks on RAG via Entailmentcited by 4
  23. 2026Hidden-in-Plain-Text: A Benchmark for Social-Web Indirect Prompt Injection in RAGcited by 3
  24. 2026Query-Efficient Agentic Graph Extraction Attacks on GraphRAG Systemscited by 3
  25. 2026Connect the Dots: Knowledge Graph-Guided Crawler Attack on Retrieval-Augmented Generation Systemscited by 3
  26. 2026InjecMEM: Memory Injection Attack on LLM Agent Memory Systemscited by 3
  27. 2024How Private are Language Models in Abstractive Summarization?cited by 2
  28. 2025Fine-Grained Privacy Extraction from Retrieval-Augmented Generation Systems via Knowledge Asymmetry Exploitationcited by 2
  29. 2025External Data Extraction Attacks against Retrieval-Augmented Large Language Modelscited by 2
  30. 2025Privacy Challenges and Solutions in Retrieval-Augmented Generation-Enhanced LLMs for Healthcare Chatbots: A Review of Applications, Risks, and Future Directionscited by 2
  31. 2026MemPot: Defending Against Memory Extraction Attack with Optimized Honeypotscited by 2
  32. 2026Differentially Private Retrieval-Augmented Generationcited by 2
  33. 2026Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directionscited by 2
  34. 2026AgenTEE: Confidential LLM Agent Execution on Edge Devicescited by 2
  35. 2026MRMMIA: Membership Inference Attacks on Memory in Chat Agentscited by 2
  36. 2026Security--Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defensecited by 2
  37. 2026Isolated but Exposed: Persistence-Based Memory Extraction Attack on LLM Agentscited by 2
  38. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Understanding Systems Trade-offs in Retrieval-Augmented Generation Model Inference
  2. 2025Token-Level Precise Attack on RAG: Searching for the Best Alternatives to Mislead Generation
  3. 2025RAGPart & RAGMask: Retrieval-Stage Defenses Against Corpus Poisoning in Retrieval-Augmented Generation
  4. 2026RAGe: A Retrieval-Augmented Generation Evaluation Framework
  5. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  6. 2024Typos that Broke the RAG's Back: Genetic Attack on RAG Pipeline by Simulating Documents in the Wild via Low-level Perturbations
  7. 2026Towards Secure Retrieval-Augmented Generation: A Comprehensive Review of Threats, Defenses and Benchmarks
  8. 2025MARAGE: Transferable Multi-Model Adversarial Attack for Retrieval-Augmented Generation Data Extraction