mlsanityData to 14 Oct 2026

Evaluating Large Language Models for Evidence-Based Clinical Question Answering

arXiv 2509.10843 ↗First posted 13 Sep 2025cs.CLcited by 1

On the map: Retrieval-augmented generation → Medical and biomedical RAG

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated substantial progress in biomedical and clinical applications, motivating rigorous evaluation of their ability to answer nuanced, evidence-based questions. We curate a multi-source benchmark drawing from Cochrane systematic reviews and clinical guidelines, including structured recommendations from the American Heart Association and narrative guidance used by insurers. Using GPT-4o-mini and GPT-5, we observe consistent performance patterns across sources and clinical domains: accuracy is highest on structured guideline recommendations (90%) and lower on narrative guideline and systematic review questions (60--70%). We also find a strong correlation between accuracy and the citation count of the underlying systematic reviews, where each doubling of citations is associated with roughly a 30% increase in the odds of a correct answer. Models show moderate ability to reason about evidence quality when contextual information is supplied. When we incorporate retrieval-augmented prompting, providing the gold-source abstract raises accuracy on previously incorrect items to 0.79; providing top 3 PubMed abstracts (ranked by semantic relevance) improves accuracy to 0.23, while random abstracts reduce accuracy (0.10, within temperature variation). These effects are mirrored in GPT-4o-mini, underscoring that source clarity and targeted retrieval -- not just model size -- drive performance. Overall, our results highlight both the promise and current limitations of LLMs for evidence-based clinical question answering. Retrieval-augmented prompting emerges as a useful strategy to improve factual accuracy and alignment with source evidence, while stratified evaluation by specialty and question type remains essential to understand current knowledge access and to contextualize model performance.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 20 of its 39 references that are in the map. First posted on arXiv 13 Sep 2025.

Builds on

20 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Benchmarking Retrieval-Augmented Generation for Medicine20 Feb
  2. 2024What Evidence Do Language Models Find Convincing?19 Feb

Other references · 18

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  3. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  4. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  5. 2023Prediction-Powered Inferencecited by 220
  6. 2025Why Language Models Hallucinatecited by 220
  7. 2024MDAgents: An Adaptive Collaboration of LLMs for Medical Decision-Makingcited by 163
  8. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 111
  9. 2023HealthFC: Verifying Health Claims with Evidence-Based Medical Fact-Checkingcited by 35
  10. 2024K-QA: A Real-World Medical Q&A Benchmarkcited by 31
  11. 2024Large Language Model Benchmarks in Medical Taskscited by 17
  12. 2024Perceptions of Linguistic Uncertainty by Language Models and Humanscited by 15
6 more
  1. 2025MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Modelscited by 10
  2. 2025AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insightscited by 10
  3. 2024MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answeringcited by 9
  4. 2025Can Large Language Models Match the Conclusions of Systematic Reviews?cited by 8
  5. 2025MIRIAD: Augmenting LLMs with millions of medical query-response pairscited by 8
  6. 2023Generating Explanations in Medical Question-Answering by Expectation Maximization Inference over Evidencecited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Contradictions in Context: Challenges for Retrieval-Augmented Generation in Healthcarecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Answering real-world clinical questions using large language model based systems
  2. 2025Introducing Answered with Evidence -- a framework for evaluating whether LLM responses to biomedical questions are founded in evidence
  3. 2025BRIDGE: Benchmarking Large Language Models for Understanding Real-world Clinical Practice Text
  4. 2025Grounding Large Language Models in Clinical Evidence: A Retrieval-Augmented Generation System for Querying UK NICE Clinical Guidelines
  5. 2026A primer on evaluation methods for large language models in healthcare
  6. 2024Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consumer Health Question Answering in Ophthalmology
  7. 2026This Treatment Works, Right? Evaluating LLM Sensitivity to Patient Question Framing in Medical QA
  8. 2025Reassessing Large Language Model Boolean Query Generation for Systematic Reviews