# MedQA-CS: Objective Structured Clinical Examination (OSCE)-Style Benchmark for Evaluating LLM Clinical Skills

arXiv: https://arxiv.org/abs/2410.01553 · HTML: https://mlsanity.com/paper/2410.01553/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2024.

## Built on by (most cited first, 17 of 17)

- [MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback](https://mlsanity.com/paper/2410.13191/) (2024)
- [RiTeK: A Dataset for Large Language Models Complex Reasoning over Textual Knowledge Graphs in Medicine](https://mlsanity.com/paper/2410.13987/) (2024)
- [DischargeSim: A Simulation Benchmark for Educational Doctor-Patient Communication at Discharge](https://mlsanity.com/paper/2509.07188/) (2025)
- [An Empirical Evaluation of Large Language Models on Consumer Health Questions](https://mlsanity.com/paper/2501.00208/) (2024)
- [ECG-Expert-QA: A Benchmark for Evaluating Medical Large Language Models in Heart Disease Diagnosis](https://mlsanity.com/paper/2502.17475/) (2025)
- [Enterprise Large Language Model Evaluation Benchmark](https://mlsanity.com/paper/2506.20274/) (2025)
- [ChatThero: An LLM-Supported Chatbot for Behavior Change and Therapeutic Support in Addiction Recovery](https://mlsanity.com/paper/2508.20996/) (2025)
- [MedCTA: A Benchmark for Clinical Tool Agents](https://mlsanity.com/paper/2606.11702/) (2026)
- [RephQA: Evaluating Readability of Large Language Models in Public Health Question Answering](https://mlsanity.com/paper/2509.16360/) (2025)
- [ChatCLIDS: Simulating Persuasive AI Dialogues to Promote Closed-Loop Insulin Adoption in Type 1 Diabetes Care](https://mlsanity.com/paper/2509.00891/) (2025)
- [AgentCaster: Reasoning-Guided Tornado Forecasting](https://mlsanity.com/paper/2510.03349/) (2025)
- [Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support](https://mlsanity.com/paper/2605.22047/) (2026)
- [HIVMedQA: Benchmarking large language models for HIV medical decision support](https://mlsanity.com/paper/2507.18143/) (2025)
- [CURE: Confidence-driven Unified Reasoning Ensemble Framework for Medical Question Answering](https://mlsanity.com/paper/2510.14353/) (2025)
- [Rethinking Patient Education as Multi-turn Multi-modal Interaction](https://mlsanity.com/paper/2604.14656/) (2026)
- [LC-ERD: Mining Latent Logic for Self-Evolving Reasoning via Consistency-Regulated Reward Decomposition](https://mlsanity.com/paper/2605.24005/) (2026)
- [SPARK: Skeleton-Guided Reasoning Synthesis from Large-Scale Scientific Literature](https://mlsanity.com/paper/2608.30214/) (2026)

## Abstract (arXiv)

Artificial intelligence (AI) and large language models (LLMs) in healthcare require advanced clinical skills (CS), yet current benchmarks fail to evaluate these comprehensively. We introduce MedQA-CS, an AI-SCE framework inspired by medical education's Objective Structured Clinical Examinations (OSCEs), to address this gap. MedQA-CS evaluates LLMs through two instruction-following tasks, LLM-as-medical-student and LLM-as-CS-examiner, designed to reflect real clinical scenarios. Our contributions include developing MedQA-CS, a comprehensive evaluation framework with publicly available data and expert annotations, and providing the quantitative and qualitative assessment of LLMs as reliable judges in CS evaluation. Our experiments show that MedQA-CS is a more challenging benchmark for evaluating clinical skills than traditional multiple-choice QA benchmarks (e.g., MedQA). Combined with existing benchmarks, MedQA-CS enables a more comprehensive evaluation of LLMs' clinical capabilities for both open- and closed-source LLMs.