# MedFactEval and MedAgentBrief: A Framework and Workflow for Generating and Evaluating Factual Clinical Summaries

arXiv: https://arxiv.org/abs/2509.05878 · HTML: https://mlsanity.com/paper/2509.05878/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 16 references that are in the map.
- First posted on arXiv 7 Sep 2025.

## Builds on (most cited first, 5 of 5)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks](https://mlsanity.com/paper/2505.23802/) (2025)
- [VeriFact: Verifying Facts in LLM-Generated Clinical Text with Electronic Health Records](https://mlsanity.com/paper/2501.16672/) (2025)
- [MedHallBench: A New Benchmark for Assessing Hallucination in Medical Large Language Models](https://mlsanity.com/paper/2412.18947/) (2024)

## Built on by (most cited first, 4 of 4)

- [CARE: A Conformal Safety Layer for Medical Summarization](https://mlsanity.com/paper/2606.08969/) (2026)
- [Monitoring Deployed AI Systems in Health Care](https://mlsanity.com/paper/2512.09048/) (2025)
- [AgenticSum: An Agentic Inference-Time Framework for Faithful Clinical Text Summarization](https://mlsanity.com/paper/2602.20040/) (2026)
- [The Verbose Context Problem in Medical Records](https://mlsanity.com/paper/2606.29503/) (2026)

## Abstract (arXiv)

Evaluating factual accuracy in Large Language Model (LLM)-generated clinical text is a critical barrier to adoption, as expert review is unscalable for the continuous quality assurance these systems require. We address this challenge with two complementary contributions. First, we introduce MedFactEval, a framework for scalable, fact-grounded evaluation where clinicians define high-salience key facts and an "LLM Jury"--a multi-LLM majority vote--assesses their inclusion in generated summaries. Second, we present MedAgentBrief, a model-agnostic, multi-step workflow designed to generate high-quality, factual discharge summaries. To validate our evaluation framework, we established a gold-standard reference using a seven-physician majority vote on clinician-defined key facts from inpatient cases. The MedFactEval LLM Jury achieved almost perfect agreement with this panel (Cohen's kappa=81%), a performance statistically non-inferior to that of a single human expert (kappa=67%, P < 0.001). Our work provides both a robust evaluation framework (MedFactEval) and a high-performing generation workflow (MedAgentBrief), offering a comprehensive approach to advance the responsible deployment of generative AI in clinical workflows.