# How well do LLMs cite relevant medical references? An evaluation framework and analyses

arXiv: https://arxiv.org/abs/2402.02008 · HTML: https://mlsanity.com/paper/2402.02008/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 42 references that are in the map.
- First posted on arXiv 3 Feb 2024.

## Builds on (most cited first, 22 of 22)

- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)
- [Teaching language models to support answers with verified quotes](https://mlsanity.com/paper/2203.11147/) (2022)
- [TrustLLM: Trustworthiness in Large Language Models](https://mlsanity.com/paper/2401.05561/) (2024)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [Med-HALT: Medical Domain Hallucination Test for Large Language Models](https://mlsanity.com/paper/2307.15343/) (2023)
- [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://mlsanity.com/paper/2401.01301/) (2024)
- [Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models](https://mlsanity.com/paper/2212.08037/) (2022)
- [Regulating ChatGPT and other Large Generative AI Models](https://mlsanity.com/paper/2302.02337/) (2023)
- [ExpertQA: Expert-Curated Questions and Attributed Answers](https://mlsanity.com/paper/2309.07852/) (2023)
- [Automatic Evaluation of Attribution by Large Language Models](https://mlsanity.com/paper/2305.06311/) (2023)
- [HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attribution](https://mlsanity.com/paper/2307.16883/) (2023)
- [Understanding Retrieval Augmentation for Long-Form Question Answering](https://mlsanity.com/paper/2310.12150/) (2023)
- [Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AI](https://mlsanity.com/paper/2311.01463/) (2023)
- [Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery](https://mlsanity.com/paper/2304.13714/) (2023)
- [Towards Verifiable Generation: A Benchmark for Knowledge-aware Language Model Attribution](https://mlsanity.com/paper/2310.05634/) (2023)

## Built on by (most cited first, 21 of 21)

- [Capabilities of Gemini Models in Medicine](https://mlsanity.com/paper/2404.18416/) (2024)
- [When Search Engine Services meet Large Language Models: Visions and Challenges](https://mlsanity.com/paper/2407.00128/) (2024)
- [From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agents](https://mlsanity.com/paper/2606.04990/) (2026)
- [MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Context](https://mlsanity.com/paper/2407.02730/) (2024)
- [FineTuneBench: How well do commercial fine-tuning APIs infuse knowledge into LLMs?](https://mlsanity.com/paper/2411.05059/) (2024)
- [MedCite: Can Language Models Generate Verifiable Text for Medicine?](https://mlsanity.com/paper/2506.06605/) (2025)
- [RadioRAG: Online Retrieval-augmented Generation for Radiology Question Answering](https://mlsanity.com/paper/2407.15621/) (2024)
- [Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality](https://mlsanity.com/paper/2506.20978/) (2025)
- [Synchronous Faithfulness Monitoring for Trustworthy Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.13692/) (2024)
- [Whose Name Comes Up? Auditing LLM-Based Scholar Recommendations](https://mlsanity.com/paper/2506.00074/) (2025)
- [Traceable LLM-based validation of statements in knowledge graphs](https://mlsanity.com/paper/2409.07507/) (2024)
- [Detecting Reference Errors in Scientific Literature with Large Language Models](https://mlsanity.com/paper/2411.06101/) (2024)
- [CurateGPT: A flexible language-model assisted biocuration tool](https://mlsanity.com/paper/2411.00046/) (2024)
- [Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consumer Health Question Answering in Ophthalmology](https://mlsanity.com/paper/2409.13902/) (2024)
- [RAG-X: Systematic Diagnosis of Retrieval-Augmented Generation for Medical Question Answering](https://mlsanity.com/paper/2603.03541/) (2026)
- [When Confidence Takes the Wrong Path: Diagnosing Retrieval-State Lock-In in RAG](https://mlsanity.com/paper/2606.22728/) (2026)
- [CryptoAnalystBench: Failures in Multi-Tool Long-Form LLM Analysis](https://mlsanity.com/paper/2602.11304/) (2026)
- [Talk2X -- An Open-Source Toolkit Facilitating Deployment of LLM-Powered Chatbots on the Web](https://mlsanity.com/paper/2504.03343/) (2025)
- [Automated Construction of a Knowledge Graph of Nuclear Fusion Energy for Effective Elicitation and Retrieval of Information](https://mlsanity.com/paper/2504.07738/) (2025)
- [The Case for Repeatable, Open, and Expert-Grounded Hallucination Benchmarks in Large Language Models](https://mlsanity.com/paper/2505.17345/) (2025)
- [Attribution Techniques for Mitigating Hallucinated Information in RAG Systems: A Survey](https://mlsanity.com/paper/2601.19927/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are currently being used to answer medical questions across a variety of clinical domains. Recent top-performing commercial LLMs, in particular, are also capable of citing sources to support their responses. In this paper, we ask: do the sources that LLMs generate actually support the claims that they make? To answer this, we propose three contributions. First, as expert medical annotations are an expensive and time-consuming bottleneck for scalable evaluation, we demonstrate that GPT-4 is highly accurate in validating source relevance, agreeing 88% of the time with a panel of medical doctors. Second, we develop an end-to-end, automated pipeline called textitSourceCheckup and use it to evaluate five top-performing LLMs on a dataset of 1200 generated questions, totaling over 40K pairs of statements and sources. Interestingly, we find that between ~50% to 90% of LLM responses are not fully supported by the sources they provide. We also evaluate GPT-4 with retrieval augmented generation (RAG) and find that, even still, around 30% of individual statements are unsupported, while nearly half of its responses are not fully supported. Third, we open-source our curated dataset of medical questions and expert annotations for future evaluations. Given the rapid pace of LLM development and the potential harms of incorrect or outdated medical information, it is crucial to also understand and quantify their capability to produce relevant, trustworthy medical references.