mlsanityData to 14 Oct 2026

Introducing Answered with Evidence -- a framework for evaluating whether LLM responses to biomedical questions are founded in evidence

arXiv 2507.02975 ↗First posted 30 Jun 2025cs.LG cs.IRcited by 0

On the map: Retrieval-augmented generation → Medical and biomedical RAG

Abstract (arXiv)

The growing use of large language models (LLMs) for biomedical question answering raises concerns about the accuracy and evidentiary support of their responses. To address this, we present Answered with Evidence, a framework for evaluating whether LLM-generated answers are grounded in scientific literature. We analyzed thousands of physician-submitted questions using a comparative pipeline that included: (1) Alexandria, fka the Atropos Evidence Library, a retrieval-augmented generation (RAG) system based on novel observational studies, and (2) two PubMed-based retrieval-augmented systems (System and Perplexity). We found that PubMed-based systems provided evidence-supported answers for approximately 44% of questions, while the novel evidence source did so for about 50%. Combined, these sources enabled reliable answers to over 70% of biomedical queries. As LLMs become increasingly capable of summarizing scientific content, maximizing their value will require systems that can accurately retrieve both published and custom-generated evidence or generate such evidence in real time.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical and biomedical RAG, one of 15 directions in Retrieval-augmented generation. Builds on 3 of its 19 references that are in the map. First posted on arXiv 30 Jun 2025.

Builds on

3 of 19 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Almanac: Retrieval-Augmented Language Models for Clinical Medicine1 Mar

Other references · 2

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2024A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Reviewcited by 66

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Large Language Models for Evidence-Based Clinical Question Answering
  2. 2024How do you know that? Teaching Generative Language Models to Reference Answers to Biomedical Questions
  3. 2025EvidenceMap: Learning Evidence Analysis to Unleash the Power of Small Language Models for Biomedical Question Answering
  4. 2024Answering real-world clinical questions using large language model based systems
  5. 2024How well do LLMs cite relevant medical references? An evaluation framework and analyses
  6. 2025M-Eval: A Heterogeneity-Based Framework for Multi-evidence Validation in Medical RAG Systems
  7. 2025Rethinking Retrieval-Augmented Generation for Medicine: A Large-Scale, Systematic Expert Evaluation and Practical Insights
  8. 2026When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG