mlsanityData to 13 Oct 2026

What Would Fix This RAG Failure? Auditing Counterfactual Response with Paired Evidence Interventions

arXiv 2608.08944 ↗First posted 9 Aug 2026cs.IR cs.LGcited by 0

On the map: Retrieval-augmented generation → RAG robustness and adaptive retrieval

Abstract (arXiv)

A failed retrieval-augmented generation (RAG) answer can be consistent with several unseen responses to evidence repair. We introduce Pair-ID, an offline audit that holds one query, retrieval state, and reader constant, then crosses two operations, adding missing support and deleting verified nonsupport, to measure a same-failure counterfactual response vector. A complete funnel over 19,981 benchmark queries identifies 11,105 eligible Qwen failures, from which a prospectively fixed SHA-256 ordering selects 1,200 before generating any sampled response. Among 1,190 regenerated-valid failures, support addition repairs 197/600 JOINT cases (0.328, 95% CI [0.292, 0.367]), and deletion repairs 162/1,190 cases (0.136, 95% CI [0.117, 0.155]); length- and position-matched shams retain semantic contrasts of 0.223 and 0.101. The original view carries partial predictive signal for individual response cells (macro AUROC 0.678; Brier 0.152 versus 0.160 for a marginal baseline), but exact-vector accuracy, 0.637, does not exceed the 0.646 majority-vector baseline, and vector macro-F1 is 0.170. Across four readers, both marginal sensitivities recur, while pooled exact-vector agreement is 0.675-0.765 and JOINT-only agreement falls to 0.538-0.691. These results show that evidence sensitivity occurs at meaningful rates in the hash-selected eligible-failure sample, is only partially predictable from the observed failure, and is conditional on the reader. The evidence supports a frame-scoped offline response audit, not an information-theoretic impossibility result, reader-independent taxonomy, or runtime repair policy.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation. Builds on 20 of its 24 references that are in the map. First posted on arXiv 9 Aug 2026.

Builds on

20 of 24 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Evidence Interfaces Shape How Retrieval-Augmented Readers Use Support19 Jul
  2. 2026Diagnosing and Repairing Factual Errors in RAG under Budget Constraints28 Jun
  3. 2026Diagnosing Evidence Utilization in Long-Context and Retrieval-Augmented Language Models under Matched Evidence Conditions4 Jun
  4. 2026Skill-RAG: Failure-State-Aware Retrieval Augmentation via Hidden-State Probing and Skill Routing17 Apr
  5. 2026CUE-R: Beyond the Final Answer in Retrieval-Augmented Generation7 Apr
  6. 2025LLM-Specific Utility for Retrieval-Augmented Generation13 Oct
  7. 2024Evidence Contextualization and Counterfactual Attribution for Conversational QA over Heterogeneous Data with RAG Systems13 Dec
  8. 2023Benchmarking Large Language Models in Retrieval-Augmented Generation4 Sep

Other references · 12

  1. 2024The Llama 3 Herd of Modelscited by 14,486
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,716
  4. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,266
  5. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,239
  6. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 829
  7. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 562
  8. 2026Doctor-RAG: A Failure-Aware Repair Framework for Agentic Retrieval-Augmented Generationcited by 5
  9. 2026When Should LLMs Search? Counterfactual Supervision for Search Routingcited by 2
  10. 2026Bridge Evidence: Static Retrieval Utility Does Not Predict Causal Utility in Multi-Step Agentic Searchcited by 2
  11. 2026When Knowledge Changes: Metamorphic Testing of RAG Systems with Mutationscited by 2
  12. 2026Improving Retrieval-Augmented Generation without Taxonomy-based Error Categorizationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RAGScope: A Leakage-Controlled, Cost-Aware Evidence-Gating Protocol for RAG Hallucination Triage
  2. 2026RECTIFY: An Interactive Workbench for Post-Evaluation RAG Diagnosis, Repair, and Verification
  3. 2026The RAT: A Unified Bayesian Model for RAG Evaluation
  4. 2026SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation
  5. 2026Assessing the Downstream Utility of Evidence-Aware Retrieval in RAG
  6. 2026Eval-Pair Matrix: Answer-Paired Meta-Evaluation of LLM Judges for Grounded RAG
  7. 2026RAG-Stress: Probing the Limits of Evidence Reliance in Retrieval-Augmented Generation
  8. 2026PURPOSE: Poisoning Conflict Resolution in RAG via Proxy-Fact-Grounded Updates