# Privacy Implications of Retrieval-Based Language Models

arXiv: https://arxiv.org/abs/2305.14888 · HTML: https://mlsanity.com/paper/2305.14888/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 51 references that are in the map.
- First posted on arXiv 24 May 2023.

## Builds on (most cited first, 30 of 30)

- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Exploiting Unintended Feature Leakage in Collaborative Learning](https://mlsanity.com/paper/1805.04049/) (2018)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [Counterfactual Memorization in Neural Language Models](https://mlsanity.com/paper/2112.12938/) (2021)
- [Training Language Models with Memory Augmentation](https://mlsanity.com/paper/2205.12674/) (2022)
- [Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipe](https://mlsanity.com/paper/2210.14348/) (2022)
- [Gmail Smart Compose: Real-Time Assisted Writing](https://mlsanity.com/paper/1906.00080/) (2019)
- [Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?](https://mlsanity.com/paper/2104.07762/) (2021)
- [Efficient Nearest Neighbor Language Models](https://mlsanity.com/paper/2109.04212/) (2021)
- [Recovering Private Text in Federated Learning of Language Models](https://mlsanity.com/paper/2205.08514/) (2022)
- [TAG: Gradient Attack on Transformer-based Language Models](https://mlsanity.com/paper/2103.06819/) (2021)
- [Selective Differential Privacy for Language Modeling](https://mlsanity.com/paper/2108.12944/) (2021)
- [LAMP: Extracting Text from Gradients with Language Model Priors](https://mlsanity.com/paper/2202.08827/) (2022)
- [Nonparametric Masked Language Modeling](https://mlsanity.com/paper/2212.01349/) (2022)
- [TextHide: Tackling Data Privacy in Language Understanding Tasks](https://mlsanity.com/paper/2010.06053/) (2020)
- [Why do Nearest Neighbor Language Models Work?](https://mlsanity.com/paper/2301.02828/) (2023)
- [Advancing Differential Privacy: Where We Are Now and Future Directions for Real-World Deployment](https://mlsanity.com/paper/2304.06929/) (2023)
- [Reasoning over Public and Private Data in Retrieval-Based Systems](https://mlsanity.com/paper/2203.11027/) (2022)

## Built on by (most cited first, 30 of 32)

- [MUSE: Machine Unlearning Six-Way Evaluation for Language Models](https://mlsanity.com/paper/2407.06460/) (2024)
- [InstructRAG: Instructing Retrieval-Augmented Generation via Self-Synthesized Rationales](https://mlsanity.com/paper/2406.13629/) (2024)
- [MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations](https://mlsanity.com/paper/2502.06453/) (2025)
- [Trustworthiness in Retrieval-Augmented Generation Systems: A Survey](https://mlsanity.com/paper/2409.10102/) (2024)
- [Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2402.17840/) (2024)
- [SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore](https://mlsanity.com/paper/2308.04430/) (2023)
- [Privacy in Large Language Models: Attacks, Defenses and Future Directions](https://mlsanity.com/paper/2310.10383/) (2023)
- [A Safe Harbor for AI Evaluation and Red Teaming](https://mlsanity.com/paper/2403.04893/) (2024)
- [Privacy Preserving Prompt Engineering: A Survey](https://mlsanity.com/paper/2404.06001/) (2024)
- [Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning](https://mlsanity.com/paper/2406.14322/) (2024)
- [AI Risk Management Should Incorporate Both Safety and Security](https://mlsanity.com/paper/2405.19524/) (2024)
- [A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage](https://mlsanity.com/paper/2504.21035/) (2025)
- [On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlook](https://mlsanity.com/paper/2307.16680/) (2023)
- [Developing Story: Case Studies of Generative AI's Use in Journalism](https://mlsanity.com/paper/2406.13706/) (2024)
- [Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs](https://mlsanity.com/paper/2512.08417/) (2025)
- [Privacy-Aware Decoding: Mitigating Privacy Leakage of Large Language Models in Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.03098/) (2025)
- [Copyright-Protected Language Generation via Adaptive Model Fusion](https://mlsanity.com/paper/2412.06619/) (2024)
- [SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2601.03979/) (2026)
- [Automated Privacy Information Annotation in Large Language Model Interactions](https://mlsanity.com/paper/2505.20910/) (2025)
- [Strong Copyright Protection for Language Models via Adaptive Model Fusion](https://mlsanity.com/paper/2407.20105/) (2024)
- [SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controlling](https://mlsanity.com/paper/2508.09105/) (2025)
- [Initial Exploration of Zero-Shot Privacy Utility Tradeoffs in Tabular Data Using GPT-4](https://mlsanity.com/paper/2404.05047/) (2024)
- [Federated Domain-Specific Knowledge Transfer on Large Language Models Using Synthetic Data](https://mlsanity.com/paper/2405.14212/) (2024)
- [External Data Extraction Attacks against Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2510.02964/) (2025)
- [Diverse And Private Synthetic Datasets Generation for RAG evaluation: A multi-agent framework](https://mlsanity.com/paper/2508.18929/) (2025)
- [Towards Privacy-Preserving Machine Translation at the Inference Stage: A New Task and Benchmark](https://mlsanity.com/paper/2603.14756/) (2026)
- [A Survey on the Real Power of ChatGPT](https://mlsanity.com/paper/2405.00704/) (2024)
- [A Case Study on the Impact of Anonymization Along the RAG Pipeline](https://mlsanity.com/paper/2604.15958/) (2026)
- [PRAG: End-to-End Privacy-Preserving Retrieval-Augmented Generation](https://mlsanity.com/paper/2604.26525/) (2026)
- [PYTHALAB-MERA: Validation-Grounded Memory, Retrieval, and Acceptance Control for Frozen-LLM Coding Agents](https://mlsanity.com/paper/2605.08468/) (2026)

## Abstract (arXiv)

Retrieval-based language models (LMs) have demonstrated improved interpretability, factuality, and adaptability compared to their parametric counterparts, by incorporating retrieved text from external datastores. While it is well known that parametric models are prone to leaking private data, it remains unclear how the addition of a retrieval datastore impacts model privacy. In this work, we present the first study of privacy risks in retrieval-based LMs, particularly kNN-LMs. Our goal is to explore the optimal design and training procedure in domains where privacy is of concern, aiming to strike a balance between utility and privacy. Crucially, we find that kNN-LMs are more susceptible to leaking private information from their private datastore than parametric models. We further explore mitigations of privacy risks. When privacy information is targeted and readily detected in the text, we find that a simple sanitization step would completely eliminate the risks, while decoupling query and key encoders achieves an even better utility-privacy trade-off. Otherwise, we consider strategies of mixing public and private data in both datastore and encoder training. While these methods offer modest improvements, they leave considerable room for future work. Together, our findings provide insights for practitioners to better understand and mitigate privacy risks in retrieval-based LMs. Our code is available at: https://github.com/Princeton-SysML/kNNLMprivacy .