# Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering

arXiv: https://arxiv.org/abs/2609.15964 · HTML: https://mlsanity.com/paper/2609.15964/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 38 references that are in the map.
- First posted on arXiv 14 Sep 2026.

## Builds on (most cited first, 17 of 17)

- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [HealthBench: Evaluating Large Language Models Towards Improved Human Health](https://mlsanity.com/paper/2505.08775/) (2025)
- [Evaluating Verifiability in Generative Search Engines](https://mlsanity.com/paper/2304.09848/) (2023)
- [Teaching language models to support answers with verified quotes](https://mlsanity.com/paper/2203.11147/) (2022)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [Almanac: Retrieval-Augmented Language Models for Clinical Medicine](https://mlsanity.com/paper/2303.01229/) (2023)
- [DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research](https://mlsanity.com/paper/2511.19399/) (2025)
- [Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models](https://mlsanity.com/paper/2212.08037/) (2022)
- [LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA](https://mlsanity.com/paper/2409.02897/) (2024)
- [Automatic Evaluation of Attribution by Large Language Models](https://mlsanity.com/paper/2305.06311/) (2023)
- [Attribute First, then Generate: Locally-attributable Grounded Text Generation](https://mlsanity.com/paper/2403.17104/) (2024)
- [Training Language Models to Generate Text with Citations via Fine-grained Rewards](https://mlsanity.com/paper/2402.04315/) (2024)
- [Search Engines in an AI Era: The False Promise of Factual and Verifiable Source-Cited Responses](https://mlsanity.com/paper/2410.22349/) (2024)
- [MedCite: Can Language Models Generate Verifiable Text for Medicine?](https://mlsanity.com/paper/2506.06605/) (2025)
- [Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries](https://mlsanity.com/paper/2606.28960/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have been widely adopted for clinical question answering (QA). Current systems can attach citations to their answers, but these often point to broad texts, leaving time-pressed clinicians unable to verify them efficiently. An alternative is to ensure that responses are verifiable by construction: providing fine-grained verbatim quotes from reference material that substantiate claims, so users can verify an answer without opening other documents. In this paper, we evaluate the ability of current models to perform this task end-to-end: from providing citations for every factual claim, to producing verbatim quotes, to ensuring that those quotes fully substantiate the claims. To do so, we build a standardized harness over four clinical practice guidelines and evaluate twelve LLMs on 222 synthetic clinical questions, measuring each of these stages separately. We find that most models can attach verbatim quotes to over 90% of their claims from prompting alone, apart from some lightweight models such as claude-haiku-4.5. Yet these quotes often fail to substantiate every detail of the claims they accompany. For instance, claude-opus-5 produces verbatim quotes for 98.0% of its claims, but fully substantiates only 37.1%. Our work provides insights into the current capability gap of LLMs in building verifiable clinical QA systems, along with artifacts for future research.