mlsanityData to 7 Oct 2026

RePCS: Diagnosing Data Memorization in LLM-Powered Retrieval-Augmented Generation

arXiv 2506.15513 ↗First posted 18 Jun 2025cs.LG cs.AIcited by 1

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Retrieval-augmented generation (RAG) has become a common strategy for updating large language model (LLM) responses with current, external information. However, models may still rely on memorized training data, bypass the retrieved evidence, and produce contaminated outputs. We introduce Retrieval-Path Contamination Scoring (RePCS), a diagnostic method that detects such behavior without requiring model access or retraining. RePCS compares two inference paths: (i) a parametric path using only the query, and (ii) a retrieval-augmented path using both the query and retrieved context by computing the Kullback-Leibler (KL) divergence between their output distributions. A low divergence suggests that the retrieved context had minimal impact, indicating potential memorization. This procedure is model-agnostic, requires no gradient or internal state access, and adds only a single additional forward pass. We further derive PAC-style guarantees that link the KL threshold to user-defined false positive and false negative rates. On the Prompt-WNQA benchmark, RePCS achieves a ROC-AUC of 0.918. This result outperforms the strongest prior method by 6.5 percentage points while keeping latency overhead below 4.7% on an NVIDIA T4 GPU. RePCS offers a lightweight, black-box safeguard to verify whether a RAG system meaningfully leverages retrieval, making it especially valuable in safety-critical applications.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 18 of its 36 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

18 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020REALM: Retrieval-Augmented Language Model Pre-Training10 Feb

Other references · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  4. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  5. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  6. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  7. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  8. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  9. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  10. 2024ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretabilitycited by 100
  11. 2024Reducing hallucination in structured outputs via Retrieval-Augmented Generationcited by 92
  12. 2024Searching for Best Practices in Retrieval-Augmented Generationcited by 83
5 more
  1. 2024Data Contamination Can Cross Language Barrierscited by 26
  2. 2024TelecomRAG: Taming Telecom Standards with Retrieval Augmented Generation and LLMscited by 13
  3. 2024Controlling Risk of Retrieval-augmented Generation: A Counterfactual Prompting Frameworkcited by 12
  4. 2024SPLATE: Sparse Late Interaction Retrievalcited by 4
  5. 2024Do You Know What You Are Talking About? Characterizing Query-Knowledge Relevance For Reliable Retrieval Augmented Generationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Revisiting RAG Retrievers: An Information Theoretic Benchmarkcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024FlashBack: Efficient Retrieval-Augmented Language Modeling for Fast Inference
  2. 2026To Memorize or to Retrieve: Scaling the Interaction Between Pretraining and Retrieval
  3. 2025Enhancing Retrieval-Augmented Generation with Two-Stage Retrieval: FlashRank Reranking and Query Expansion
  4. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence
  5. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
  6. 2026Reason and Verify: A Framework for Faithful Retrieval-Augmented Generation
  7. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models
  8. 2025PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation