# Mask-based Membership Inference Attacks for Retrieval-Augmented Generation

arXiv: https://arxiv.org/abs/2410.20142 · HTML: https://mlsanity.com/paper/2410.20142/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 26 Oct 2024.

## Placed in

- Retrieval-augmented generation / [Privacy and security of RAG](https://mlsanity.com/rag/privacy-and-security-of-rag/)
- LLM safety and security / [Membership inference attacks](https://mlsanity.com/safety/membership-inference-attacks/)

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023)
- [Precise Zero-Shot Dense Retrieval without Relevance Labels](https://mlsanity.com/paper/2212.10496/) (2022)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [Membership Inference Attacks on Machine Learning: A Survey](https://mlsanity.com/paper/2103.07853/) (2021)
- [Membership Inference Attacks against Language Models via Neighbourhood Comparison](https://mlsanity.com/paper/2305.18462/) (2023)
- [The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)](https://mlsanity.com/paper/2402.16893/) (2024)
- [Certifiably Robust RAG against Retrieval Corruption](https://mlsanity.com/paper/2405.15556/) (2024)
- [Foundation Models and Fair Use](https://mlsanity.com/paper/2303.15715/) (2023)
- [Trustworthiness in Retrieval-Augmented Generation Systems: A Survey](https://mlsanity.com/paper/2409.10102/) (2024)
- [Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2402.17840/) (2024)
- [UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation](https://mlsanity.com/paper/2303.08518/) (2023)
- [Augmentation-Adapted Retriever Improves Generalization of Language Models as Generic Plug-In](https://mlsanity.com/paper/2305.17331/) (2023)
- [SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore](https://mlsanity.com/paper/2308.04430/) (2023)
- [Right to be Forgotten in the Era of Large Language Models: Implications, Challenges, and Solutions](https://mlsanity.com/paper/2307.03941/) (2023)
- [Knowledge Graph-Augmented Language Models for Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2305.18846/) (2023)
- [Mask and Cloze: Automatic Open Cloze Question Generation using a Masked Language Model](https://mlsanity.com/paper/2205.07202/) (2022)

## Built on by (most cited first, 29 of 29)

- [Membership Inference Attacks on Large-Scale Models: A Survey](https://mlsanity.com/paper/2503.19338/) (2025)
- [RAG Security and Privacy: Formalizing the Threat Model and Attack Surface](https://mlsanity.com/paper/2509.20324/) (2025)
- [Position: LLM Watermarking Should Align Stakeholders' Incentives for Practical Adoption](https://mlsanity.com/paper/2510.18333/) (2025)
- [SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2601.03979/) (2026)
- [Differentially Private Synthetic Text Generation for Retrieval-Augmented Generation (RAG)](https://mlsanity.com/paper/2510.06719/) (2025)
- [ImageSentinel: Protecting Visual Datasets from Unauthorized Retrieval-Augmented Image Generation](https://mlsanity.com/paper/2510.12119/) (2025)
- [Five Queries Are Enough: Query-Efficient and Surrogate-Free Membership Inference Attacks on RAG via Entailment](https://mlsanity.com/paper/2605.24312/) (2026)
- [SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controlling](https://mlsanity.com/paper/2508.09105/) (2025)
- [Private-RAG: Answering Multiple Queries with LLMs while Keeping Your Data Private](https://mlsanity.com/paper/2511.07637/) (2025)
- [Privacy Challenges and Solutions in Retrieval-Augmented Generation-Enhanced LLMs for Healthcare Chatbots: A Review of Applications, Risks, and Future Directions](https://mlsanity.com/paper/2511.11347/) (2025)
- [BudgetLeak: Membership Inference Attacks on RAG Systems via the Generation Budget Side Channel](https://mlsanity.com/paper/2511.12043/) (2025)
- [Connect the Dots: Knowledge Graph-Guided Crawler Attack on Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2601.15678/) (2026)
- [Distributed Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.00443/) (2025)
- [MrM: Black-Box Membership Inference Attacks against Multimodal RAG Systems](https://mlsanity.com/paper/2506.07399/) (2025)
- [Fine-Grained Privacy Extraction from Retrieval-Augmented Generation Systems via Knowledge Asymmetry Exploitation](https://mlsanity.com/paper/2507.23229/) (2025)
- [RedVisor: Reasoning-Aware Prompt Injection Defense via Zero-Copy KV Cache Reuse](https://mlsanity.com/paper/2602.01795/) (2026)
- [MRMMIA: Membership Inference Attacks on Memory in Chat Agents](https://mlsanity.com/paper/2605.27825/) (2026)
- [KEPo: Knowledge Evolution Poison on Graph-based Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.11501/) (2026)
- [Sensitivity-Aware Retrieval-Augmented Intent Clarification](https://mlsanity.com/paper/2603.06025/) (2026)
- [Automated Membership Inference Attacks (AutoMIA): Discovering MIA Signal Computations using LLM Agents](https://mlsanity.com/paper/2603.19375/) (2026)
- [Adaptive Defense Orchestration for RAG: A Sentinel-Strategist Architecture against Multi-Vector Attacks](https://mlsanity.com/paper/2604.20932/) (2026)
- [E-MIA: Exam-Style Black-Box Membership Inference Attacks against RAG Systems](https://mlsanity.com/paper/2605.00955/) (2026)
- [On the Privacy of LLMs: An Ablation Study](https://mlsanity.com/paper/2605.02255/) (2026)
- [ImageAuditor: Membership Inference Attack against Image-based Retrieval-Augmented Generation](https://mlsanity.com/paper/2606.03354/) (2026)
- [Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systems](https://mlsanity.com/paper/2606.25533/) (2026)
- [Privacy-Preserving RAG by Concealing Sensitive Information from External LLMs](https://mlsanity.com/paper/2608.12675/) (2026)
- [Context Inference Attacks Without Jailbreaks](https://mlsanity.com/paper/2609.01663/) (2026)
- [K-Bench: A Benchmark for LLM Unlearning in Agentic Deployments](https://mlsanity.com/paper/2609.12808/) (2026)
- [CIG-MIA: Context-Induced Information Gain Membership Inference Attacks against Retrieval-Augmented Generation](https://mlsanity.com/paper/2609.14649/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has been an effective approach to mitigate hallucinations in large language models (LLMs) by incorporating up-to-date and domain-specific knowledge. Recently, there has been a trend of storing up-to-date or copyrighted data in RAG knowledge databases instead of using it for LLM training. This practice has raised concerns about Membership Inference Attacks (MIAs), which aim to detect if a specific target document is stored in the RAG system's knowledge database so as to protect the rights of data producers. While research has focused on enhancing the trustworthiness of RAG systems, existing MIAs for RAG systems remain largely insufficient. Previous work either relies solely on the RAG system's judgment or is easily influenced by other documents or the LLM's internal knowledge, which is unreliable and lacks explainability. To address these limitations, we propose a Mask-Based Membership Inference Attacks (MBA) framework. Our framework first employs a masking algorithm that effectively masks a certain number of words in the target document. The masked text is then used to prompt the RAG system, and the RAG system is required to predict the mask values. If the target document appears in the knowledge database, the masked text will retrieve the complete target document as context, allowing for accurate mask prediction. Finally, we adopt a simple yet effective threshold-based method to infer the membership of target document by analyzing the accuracy of mask prediction. Our mask-based approach is more document-specific, making the RAG system's generation less susceptible to distractions from other documents or the LLM's internal knowledge. Extensive experiments demonstrate the effectiveness of our approach compared to existing baseline models.