# Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines

arXiv: https://arxiv.org/abs/2609.24122 · HTML: https://mlsanity.com/paper/2609.24122/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation.
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 21 Sep 2026.

## Placed in

- Retrieval-augmented generation / [RAG pipeline design and evaluation](https://mlsanity.com/rag/rag-pipeline-design-and-evaluation/)

## Builds on (most cited first, 26 of 26)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://mlsanity.com/paper/2212.10509/) (2022)
- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [Active Retrieval Augmented Generation](https://mlsanity.com/paper/2305.06983/) (2023)
- [Benchmarking Large Language Models in Retrieval-Augmented Generation](https://mlsanity.com/paper/2309.01431/) (2023)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2311.09476/) (2023)
- [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries](https://mlsanity.com/paper/2401.15391/) (2024)
- [Document Expansion by Query Prediction](https://mlsanity.com/paper/1904.08375/) (2019)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [Perspectives on Large Language Models for Relevance Judgment](https://mlsanity.com/paper/2304.09161/) (2023)
- [Don't Use LLMs to Make Relevance Judgments](https://mlsanity.com/paper/2409.15133/) (2024)
- [TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection](https://mlsanity.com/paper/2005.04474/) (2020)
- [A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look](https://mlsanity.com/paper/2411.08275/) (2024)
- [LLMs Can Patch Up Missing Relevance Judgments in Evaluation](https://mlsanity.com/paper/2405.04727/) (2024)
- [Controlled Retrieval-augmented Context Evaluation for Long-form RAG](https://mlsanity.com/paper/2506.20051/) (2025)
- [Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coverage](https://mlsanity.com/paper/2410.15531/) (2024)
- [LANCER: LLM Reranking for Nugget Coverage](https://mlsanity.com/paper/2601.22008/) (2026)
- [CoverageBench: Evaluating Information Coverage across Tasks and Domains](https://mlsanity.com/paper/2603.20034/) (2026)

## Abstract (arXiv)

Retrieval-augmented generation (RAG) is hard to monitor in production: exhaustive relevance labels do not exist for non-stationary multi-million-passage corpora that re-index in real time. As a result, retrieval quality is generally understudied and often deprioritised in favour of generation-oriented metrics. In this work, we propose auditing retrieval coverage by probing for evidence of missing documents rather than enumerating every relevant one. Our method Re:CAP (REtrieval Coverage Audit by iterative Probing) is a reference-free audit loop applied to a deployed RAG pipeline's initial answer and retrieved context: it identifies the topics already covered, generates probing questions for plausibly missing topics, retrieves candidate documents, and applies an LLM-as-judge to retain only those that introduce previously-unretrieved information. On four public benchmarks, Re:CAP recovers 9-29% of gold labels that flat BM25 top-500 cannot reach, rising to 48% on TREC-COVID. On MuSiQue Re:CAP beats flat hybrid top-500 by +12.9 pp on recall at less than half the document budget. An ensemble BM25, dense, and hybrid baseline (top-500 each) still leaves out 21.2% of gold docs on TREC-COVID that Re:CAP recovers; human annotators judge that 78.9% of those structurally distinct documents add new information to the baseline answer (Fleiss kappa = 0.79, n = 123), and 73.9% on live production traffic (n = 180). End-to-end recall is reproducible to within ±1% across three independent runs, making Re:CAP a stable instrument for periodic retrieval audits.