mlsanityData to 14 Oct 2026

When Retrieval Helps and Distracts: Evaluating Evidence-Generating LLMs for Biomedical Claim Verification

arXiv 2608.01409 ↗First posted 2 Aug 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Biomedical fact-checking systems must do more than predict whether a claim is supported, contradicted, or unaddressed: they should also produce evidence that is faithful, complete, and useful for verification. We study this evidence-generation setting on CARE-XAI, a unified benchmark spanning five biomedical and health fact-checking sources. We compare base instruction LLMs, PubMed retrieval-augmented LLMs, fine-tuned LLMs, label-only LLMs, and biomedical encoder classifiers under a shared evaluation protocol. Biomedical classifiers remain strongest for verdict-only prediction, while fine-tuned LLMs are the strongest evidence-generating systems. PubMed retrieval is mixed: it helps PubMed-aligned sources such as PubMedQA and SciFact, but can distract models on broader public-health claims. We introduce Bio-GRACE, a gold-reference-normalized diagnostic for measuring whether retrieved evidence recovers the decision benefit of reference evidence. Bio-GRACE shows that retrieval utility is source-dependent, motivates selective retrieval, and exposes why retrieval recall and lexical evidence overlap are insufficient for biomedical fact-checking.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 50 references that are in the map. First posted on arXiv 2 Aug 2026.

Builds on

33 of 50 references.

References · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  10. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  11. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  12. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
21 more
  1. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  2. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  3. 2017Selective Classification for Deep Neural Networkscited by 852
  4. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  5. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  6. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  7. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  8. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
  9. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  10. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 369
  11. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
  12. 2026Gemma 4 Technical Reportcited by 259
  13. 2020Explainable Automated Fact-Checking for Public Health Claimscited by 211
  14. 2019MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claimscited by 169
  15. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 151
  16. 2023HealthFC: Verifying Health Claims with Evidence-Based Medical Fact-Checkingcited by 35
  17. 2025Worse than Zero-shot? A Fact-Checking Dataset for Evaluating the Robustness of RAG Against Misleading Retrievalscited by 28
  18. 2021MedNLI Is Not Immune: Natural Language Inference Artifacts in the Clinical Domaincited by 17
  19. 2025Combining Evidence and Reasoning for Biomedical Fact-Checkingcited by 3
  20. 2025MedBioRAG: Semantic Search and Retrieval-Augmented Generation with Large Language Models for Medical and Biological QAcited by 1
  21. 2026VerifAI: A Verifiable Open-Source Search Engine for Biomedical Question Answeringcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MedRAGChecker: Claim-Level Verification for Biomedical Retrieval-Augmented Generation
  2. 2025Combating Biomedical Misinformation through Multi-modal Claim Detection and Evidence-based Verification
  3. 2025Explainable Biomedical Claim Verification with Large Language Models
  4. 2026CLEAR: Cross-Source Evidence Adjudication for Large Language Models in Medicine
  5. 2025Introducing Answered with Evidence -- a framework for evaluating whether LLM responses to biomedical questions are founded in evidence
  6. 2026PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering
  7. 2025MedTrust-RAG: Evidence Verification and Trust Alignment for Biomedical Question Answering
  8. 2025Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights