mlsanityData to 7 Oct 2026

Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines

arXiv 2609.24122 ↗First posted 21 Sep 2026cs.CLcited by 0

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation also fits: RAG robustness and adaptive retrieval

Abstract (arXiv)

Retrieval-augmented generation (RAG) is hard to monitor in production: exhaustive relevance labels do not exist for non-stationary multi-million-passage corpora that re-index in real time. As a result, retrieval quality is generally understudied and often deprioritised in favour of generation-oriented metrics. In this work, we propose auditing retrieval coverage by probing for evidence of missing documents rather than enumerating every relevant one. Our method Re:CAP (REtrieval Coverage Audit by iterative Probing) is a reference-free audit loop applied to a deployed RAG pipeline's initial answer and retrieved context: it identifies the topics already covered, generates probing questions for plausibly missing topics, retrieves candidate documents, and applies an LLM-as-judge to retain only those that introduce previously-unretrieved information. On four public benchmarks, Re:CAP recovers 9-29% of gold labels that flat BM25 top-500 cannot reach, rising to 48% on TREC-COVID. On MuSiQue Re:CAP beats flat hybrid top-500 by +12.9 pp on recall at less than half the document budget. An ensemble BM25, dense, and hybrid baseline (top-500 each) still leaves out 21.2% of gold docs on TREC-COVID that Re:CAP recovers; human annotators judge that 78.9% of those structurally distinct documents add new information to the baseline answer (Fleiss kappa = 0.79, n = 123), and 73.9% on live production traffic (n = 180). End-to-end recall is reproducible to within ±1% across three independent runs, making Re:CAP a stable instrument for periodic retrieval audits.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 26 of its 39 references that are in the map. First posted on arXiv 21 Sep 2026.

Builds on

26 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Controlled Retrieval-augmented Context Evaluation for Long-form RAG24 Jun
  2. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries27 Jan
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec
  4. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 22

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  6. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  8. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 911
  9. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  10. 2023Active Retrieval Augmented Generationcited by 506
  11. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  12. 2020Overview of the TREC 2019 deep learning trackcited by 282
10 more
  1. 2019Document Expansion by Query Predictioncited by 225
  2. 2023Large language models can accurately predict searcher preferencescited by 129
  3. 2023Perspectives on Large Language Models for Relevance Judgmentcited by 84
  4. 2024Don't Use LLMs to Make Relevance Judgmentscited by 26
  5. 2020TREC-COVID: Constructing a Pandemic Information Retrieval Test Collectioncited by 23
  6. 2024A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Lookcited by 18
  7. 2024LLMs Can Patch Up Missing Relevance Judgments in Evaluationcited by 12
  8. 2024Do RAG Systems Cover What Matters? Evaluating and Optimizing Responses with Sub-Question Coveragecited by 8
  9. 2026LANCER: LLM Reranking for Nugget Coveragecited by 2
  10. 2026CoverageBench: Evaluating Information Coverage across Tasks and Domainscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Stronger Baselines for Retrieval-Augmented Generation with Long-Context Language Models
  2. 2026Testing Retrieval-Augmented Generation Systems with Chunk Coverage
  3. 2026From BM25 to Corrective RAG: Benchmarking Retrieval Strategies for Text-and-Table Documents
  4. 2026Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage
  5. 2024Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall
  6. 2026M-RAG: Semantic Key-Value Indexing for Retrieval-Augmented Generation
  7. 2026Training the Knowledge Base through Evidence Distillation and Write-Back Enrichment
  8. 2026Scaling Retrieval Augmented Generation with RAG Fusion: Lessons from an Industry Deployment