mlsanityData to 14 Oct 2026

Same Question, Different Source, Different Answer: Auditing Source-Dependence in Medical Multi-Source RAG

arXiv 2605.29084 ↗First posted 27 May 2026cs.CL cs.AI cs.IRcited by 0

On the map: Retrieval-augmented generation → Medical and biomedical RAG

Abstract (arXiv)

A retrieval-augmented generation (RAG) system deployed over a multi-author institutional corpus can give a different answer to the same question depending on which source it retrieves -- a failure mode the dominant single-gold-answer paradigm cannot diagnose. We argue that source-dependence is a missing axis of NLP evaluation, and that auditing it means shifting the unit of evaluation from answer correctness to the inter-source relationship. We make this concrete in transplant patient education, where institutional sources demonstrably disagree, releasing three artefacts: TransplantQA, a benchmark of real patient questions, each answered by grounding generation in multiple institutional handbooks as candidate sources; HERO-QA, a hierarchical retrieval strategy that grounds and audits each answer; and a structured-output judge that scores inter-source relationships on a validated 5-label taxonomy. At scale, better retrieval reveals far more disagreement than prior estimates suggested -- understating its prevalence, not its intensity. The framework is domain-agnostic and transfers to legal and educational RAG: measuring source-dependence is a responsibility for deployed multi-source NLP generally.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 13 of its 23 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

13 of 23 references.

References · 13

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  5. 2018Datasheets for Datasetscited by 1,386
  6. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  7. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  9. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
  10. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  11. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
  12. 2020MedDialog: Two Large-scale Medical Dialogue Datasetscited by 161
1 more
  1. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026When Documents Disagree: Measuring Institutional Variation in Transplant Guidance with Retrieval-Augmented Language Models
  2. 2025Contradictions in Context: Challenges for Retrieval-Augmented Generation in Healthcare
  3. 2026When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG
  4. 2025M-Eval: A Heterogeneity-Based Framework for Multi-evidence Validation in Medical RAG Systems
  5. 2025Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights
  6. 2024How well do LLMs cite relevant medical references? An evaluation framework and analyses
  7. 2025Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domain
  8. 2024Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consumer Health Question Answering in Ophthalmology