# Mind the Hook: Source-Level Auditing of Privacy Defenses in Retrieval-Augmented Generation

arXiv: https://arxiv.org/abs/2608.09001 · HTML: https://mlsanity.com/paper/2608.09001/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 15 of its 16 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Placed in

- Retrieval-augmented generation / [Privacy and security of RAG](https://mlsanity.com/rag/privacy-and-security-of-rag/)

## Builds on (most cited first, 15 of 15)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)](https://mlsanity.com/paper/2402.16893/) (2024)
- [Privacy-Preserving Retrieval-Augmented Generation with Differential Privacy](https://mlsanity.com/paper/2412.04697/) (2024)
- [Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applications](https://mlsanity.com/paper/2411.14110/) (2024)
- [SafetyRepro: Configuration-Conditional Rank Instability on Alignment Benchmarks](https://mlsanity.com/paper/2605.25492/) (2026)
- [Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Audit](https://mlsanity.com/paper/2605.28873/) (2026)
- [Auditing Reasoning-Trace Memorization Claims after Unlearning with Head-Conditioned Canaries](https://mlsanity.com/paper/2605.18891/) (2026)
- [Auditing and Fixing Economic Validity in Tabular Foundation Models for Discrete Choice](https://mlsanity.com/paper/2605.26559/) (2026)

## Abstract (arXiv)

Black-box privacy scores for retrieval-augmented generation (RAG) are difficult to interpret unless the audited defense's active pipeline hook is known. We propose an active-path audit: inventory source-level hooks over retrieval, retrieved content, and generation; map each metric to the leakage channel it observes; and validate generated-text effects with exact-match canaries. In our benchmark reimplementations, the DP-style defenses modify retrieval scores only: their generation hooks are TODO-flagged stubs that return responses unchanged. This active path explains why they affect membership-inference behavior but track No-Defense on generated-text named-entity leakage, measured by NEL_strict. By contrast, the end-to-end LPRAG path is canary-validated on the email channel, recovering 53/150 canaries under No-Defense and 0/150 under LPRAG. These findings concern our reimplementations on our stack, not released defenses or defense families; the contribution is a methodology and case study, not a universal ranking