mlsanityData to 7 Oct 2026

Privacy-Preserving Retrieval-Augmented Generation with Differential Privacy

arXiv 2412.04697 ↗First posted 6 Dec 2024cs.CR cs.AI cs.CLcited by 38

On the map: Retrieval-augmented generation → Privacy and security of RAG

Abstract (arXiv)

With the recent remarkable advancement of large language models (LLMs), there has been a growing interest in utilizing them in the domains with highly sensitive data that lies outside their training data. For this purpose, retrieval-augmented generation (RAG) is particularly effective -- it assists LLMs by directly providing relevant information from the external knowledge sources. However, without extra privacy safeguards, RAG outputs risk leaking sensitive information from the external data source. In this work, we explore RAG under differential privacy (DP), a formal guarantee of data privacy. The main challenge with differentially private RAG is how to generate long accurate answers within a moderate privacy budget. We address this by proposing an algorithm that smartly spends privacy budget only for the tokens that require the sensitive information and uses the non-private LLM for other tokens. Our extensive empirical evaluations reveal that our algorithm outperforms the non-RAG baseline under a reasonable privacy budget of ε≈ 10 across different models and datasets.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation. Builds on 34 of its 43 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

34 of 43 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Data Extraction Attacks in Retrieval-Augmented Generation via Backdoors3 Nov
  2. 2024Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Data20 Jun
  3. 2024Phantom: General Backdoor Attacks on Retrieval Augmented Language Generation30 May
  4. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systems27 Feb
  5. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)23 Feb

Other references · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2016Deep Learning with Differential Privacycited by 2,682
  9. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  12. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
17 more
  1. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  2. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  3. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  4. 2013The Composition Theorem for Differential Privacycited by 290
  5. 2018Scalable Private Learning with PATEcited by 266
  6. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 94
  7. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 80
  8. 2020Individual Privacy Accounting via a Renyi Filtercited by 57
  9. 2023DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineercited by 56
  10. 2023Privacy-Preserving In-Context Learning for Large Language Modelscited by 50
  11. 2019Practical Differentially Private Top-k Selection with Pay-what-you-get Compositioncited by 38
  12. 2016Privacy Odometers and Filters: Pay-as-you-Go Compositioncited by 32
  13. 2023Joint Prompt Optimization of Stacked LLMs using Variational Inferencecited by 32
  14. 2024RAGGED: Towards Informed Design of Scalable and Stable RAG Systemscited by 24
  15. 2022Fully Adaptive Composition in Differential Privacycited by 22
  16. 2024RetrievalQA: Assessing Adaptive Retrieval-Augmented Generation for Short-form Open-Domain Question Answeringcited by 13
  17. 2021Practical Privacy Filters and Odometers with R'enyi Differential Privacy and Applications to Differentially Private Deep Learningcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 15; 2026: 22.

38 papers.

Sort:

By area:

  1. 2024RemoteRAG: A Privacy-Preserving LLM Cloud RAG Servicecited by 17
  2. 2025Safeguarding Privacy of Retrieval Data against Membership Inference Attacks: Is This Query Too Close to Home?cited by 8
  3. 2025Privacy-Aware Decoding: Mitigating Privacy Leakage of Large Language Models in Retrieval-Augmented Generationcited by 8
  4. 2025RAG Security and Privacy: Formalizing the Threat Model and Attack Surfacecited by 8
  5. 2026MemPrivacy: Privacy-Preserving Personalized Memory Management for Edge-Cloud Agentscited by 7
  6. 2025Evaluating Differentially Private Generation of Domain-Specific Textcited by 5
  7. 2025Differentially Private Synthetic Text Generation for Retrieval-Augmented Generation (RAG)cited by 5
  8. 2025Exposing Privacy Risks in Graph Retrieval-Augmented Generationcited by 4
  9. 2025SynBench: A Benchmark for Differentially Private Text Generationcited by 4
  10. 2026Efficient Privacy-Preserving Retrieval Augmented Generation with Distance-Preserving Encryptioncited by 4
  11. 2026Observable Channels, Not Just Storage: Evaluating Privacy Leakage in LLM Agent Pipelinescited by 4
  12. 2025A Comprehensive Review on Harnessing Large Language Models to Overcome Recommender System Challengescited by 3
26 more
  1. 2025SMA: Who Said That? Auditing Membership Leakage in Semi-Black-box RAG Controllingcited by 3
  2. 2026Towards Secure Retrieval-Augmented Generation: A Comprehensive Review of Threats, Defenses and Benchmarkscited by 3
  3. 2025Distributed Retrieval-Augmented Generationcited by 2
  4. 2025Private-RAG: Answering Multiple Queries with LLMs while Keeping Your Data Privatecited by 2
  5. 2025Privacy Challenges and Solutions in Retrieval-Augmented Generation-Enhanced LLMs for Healthcare Chatbots: A Review of Applications, Risks, and Future Directionscited by 2
  6. 2026Differentially Private Retrieval-Augmented Generationcited by 2
  7. 2026Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directionscited by 2
  8. 2025Cocoon: A System Architecture for Differentially Private Training with Correlated Noisescited by 1
  9. 2025Differentially Private In-Context Learning with Nearest Neighbor Searchcited by 1
  10. 2026P2RAG: Efficient Privacy-Preserving RAG Service Supporting Arbitrary Top-k Retrievalcited by 1
  11. 2026Differential Privacy in Generative AI Agents: Analysis and Optimal Tradeoffscited by 1
  12. 2026ToolMinimize: Auditing and Rewriting LLM Agent Tool Calls to Minimize Privacy Exposurecited by 1
  13. 2025Prior Knowledge Makes It Possible: From Sublinear Graph Algorithms to LLM Test-Time Methodscited by 0
  14. 2026SEAL-Tag: Self-Tag Evidence Aggregation with Probabilistic Circuits for PII-Safe Retrieval-Augmented Generationcited by 0
  15. 2026PRAG: End-to-End Privacy-Preserving Retrieval-Augmented Generationcited by 0
  16. 2026ImageAuditor: Membership Inference Attack against Image-based Retrieval-Augmented Generationcited by 0
  17. 2026Privacy-Preserving RAG via Multi-Agent Semantic Rewriting: Achieving Confidentiality Without Compromising Contextual Fidelitycited by 0
  18. 2026Agents That Know Too Much: A Data-Centric Survey of Privacy in LLM Agentscited by 0
  19. 2026Is External Database Protection Static in Retrieval-Augmented Generation? Rethinking Privacy Preservation under Dynamic Queriescited by 0
  20. 2026TriShieldRAG: 3 Rings, One Blind Spot in Layered Defenses for Retrieval-Augmented Generationcited by 0
  21. 2026DP-MemView: A Memory Interface for Attribute-Level Transcript Privacy in Long-Term LLM Agentscited by 0
  22. 2026Mind the Hook: Source-Level Auditing of Privacy Defenses in Retrieval-Augmented Generationcited by 0
  23. 2026Mitigating Database Leakage in RAG Systems with Keyword-Grounded Fact Substitutioncited by 0
  24. 2026The Privacy-Hallucination Tradeoff in Differentially Private Language Modelscited by 0
  25. 2026CIPL: A Channel-Aware Framework for Recoverable Privacy Leakage in LLM Agentscited by 0
  26. 2026Only Pay What You Must Spend: On-Demand Privacy Budget Payment for Differentially Private RAGcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RAG with Differential Privacy
  2. 2026SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systems
  3. 2025Privacy-Aware RAG: Secure and Isolated Knowledge Retrieval
  4. 2026Not All Entities are Created Equal: A Dynamic Anonymization Framework for Privacy-Preserving RAG
  5. 2024Generating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generation
  6. 2026Security and Privacy in Retrieval-Augmented Generation: Architectures, Threats, Defenses, and Future Directions for Building Trustworthy Systems
  7. 2024Pirates of the RAG: Adaptively Attacking LLMs to Leak Knowledge Bases
  8. 2025Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generation