# Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm

arXiv: https://arxiv.org/abs/2403.16446 · HTML: https://mlsanity.com/paper/2403.16446/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 25 Mar 2024.

## Builds on (most cited first, 9 of 9)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [How Language Model Hallucinations Can Snowball](https://mlsanity.com/paper/2305.13534/) (2023)
- [HuatuoGPT, towards Taming Language Model to Be a Doctor](https://mlsanity.com/paper/2305.15075/) (2023)
- [HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs](https://mlsanity.com/paper/2311.09774/) (2023)
- [LLM-Mini-CEX: Automatic Evaluation of Large Language Model for Diagnostic Conversation](https://mlsanity.com/paper/2308.07635/) (2023)
- [From Beginner to Expert: Modeling Medical Knowledge into General LLMs](https://mlsanity.com/paper/2312.01040/) (2023)

## Built on by (most cited first, 14 of 14)

- [A Survey of LLM-based Agents in Medicine: How far are we from Baymax?](https://mlsanity.com/paper/2502.11211/) (2025)
- [A Survey on Medical Large Language Models: Technology, Application, Trustworthiness, and Future Directions](https://mlsanity.com/paper/2406.03712/) (2024)
- [A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domains](https://mlsanity.com/paper/2507.23486/) (2025)
- [Automating Expert-Level Medical Reasoning Evaluation of Large Language Models](https://mlsanity.com/paper/2507.07988/) (2025)
- [ERVQA: A Dataset to Benchmark the Readiness of Large Vision Language Models in Hospital Environments](https://mlsanity.com/paper/2410.06420/) (2024)
- [MedConsultBench: A Full-Cycle, Fine-Grained, Process-Aware Benchmark for Medical Consultation Agents](https://mlsanity.com/paper/2601.12661/) (2026)
- [RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignment](https://mlsanity.com/paper/2408.12579/) (2024)
- [Improving Interactive Diagnostic Ability of a Large Language Model Agent Through Clinical Experience Learning](https://mlsanity.com/paper/2503.16463/) (2025)
- [Retrospective Comparative Analysis of Prostate Cancer In-Basket Messages: Responses from Closed-Domain LLM vs. Clinical Teams](https://mlsanity.com/paper/2409.18290/) (2024)
- [Med-CoDE: Medical Critique based Disagreement Evaluation Framework](https://mlsanity.com/paper/2504.15330/) (2025)
- [MedSyn: Enhancing Diagnostics with Human-AI Collaboration](https://mlsanity.com/paper/2506.14774/) (2025)
- [MedSynth: Realistic, Synthetic Medical Dialogue-Note Pairs](https://mlsanity.com/paper/2508.01401/) (2025)
- [Note2Chat: Improving LLMs for Multi-Turn Clinical History Taking Using Medical Notes](https://mlsanity.com/paper/2601.21551/) (2026)
- [Human-LLM Dialogue Improves Diagnostic Accuracy in Emergency Care](https://mlsanity.com/paper/2605.08533/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are gaining increasing interests to improve clinical efficiency for medical diagnosis, owing to their unprecedented performance in modelling natural language. Ensuring the safe and reliable clinical applications, the evaluation of LLMs indeed becomes critical for better mitigating the potential risks, e.g., hallucinations. However, current evaluation methods heavily rely on labor-intensive human participation to achieve human-preferred judgements. To overcome this challenge, we propose an automatic evaluation paradigm tailored to assess the LLMs' capabilities in delivering clinical services, e.g., disease diagnosis and treatment. The evaluation paradigm contains three basic elements: metric, data, and algorithm. Specifically, inspired by professional clinical practice pathways, we formulate a LLM-specific clinical pathway (LCP) to define the clinical capabilities that a doctor agent should possess. Then, Standardized Patients (SPs) from the medical education are introduced as the guideline for collecting medical data for evaluation, which can well ensure the completeness of the evaluation procedure. Leveraging these steps, we develop a multi-agent framework to simulate the interactive environment between SPs and a doctor agent, which is equipped with a Retrieval-Augmented Evaluation (RAE) to determine whether the behaviors of a doctor agent are in accordance with LCP. The above paradigm can be extended to any similar clinical scenarios to automatically evaluate the LLMs' medical capabilities. Applying such paradigm, we construct an evaluation benchmark in the field of urology, including a LCP, a SPs dataset, and an automated RAE. Extensive experiments are conducted to demonstrate the effectiveness of the proposed approach, providing more insights for LLMs' safe and reliable deployments in clinical practice.