# LLM-Based Data Generation and Clinical Skills Evaluation for Low-Resource French OSCEs

arXiv: https://arxiv.org/abs/2604.08126 · HTML: https://mlsanity.com/paper/2604.08126/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 24 references that are in the map.
- First posted on arXiv 9 Apr 2026.

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [MedDialog: Two Large-scale Medical Dialogue Datasets](https://mlsanity.com/paper/2004.03329/) (2020)
- [Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates](https://mlsanity.com/paper/2408.13006/) (2024)
- [DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains](https://mlsanity.com/paper/2304.00958/) (2023)
- [Synthetic Patient-Physician Dialogue Generation from Clinical Notes Using LLM](https://mlsanity.com/paper/2408.06285/) (2024)
- [Benchmarking Generative AI for Scoring Medical Student Interviews in Objective Structured Clinical Examinations (OSCEs)](https://mlsanity.com/paper/2501.13957/) (2025)
- [Large Language Models for Medical OSCE Assessment: A Novel Approach to Transcript Analysis](https://mlsanity.com/paper/2410.12858/) (2024)

## Built on by (most cited first, 1 of 1)

- [A French OSCE Dialogue Dataset and Controllable Virtual Patient System for Clinical Training](https://mlsanity.com/paper/2606.28526/) (2026)

## Abstract (arXiv)

Objective Structured Clinical Examinations (OSCEs) are the standard method for assessing medical students' clinical and communication skills through structured patient interviews. In France, however, the organization of training sessions is limited by human and logistical constraints, restricting students' access to repeated practice and structured feedback. Recent advances in Natural Language Processing (NLP) and Large Language Models (LLMs) now offer the opportunity to automatically evaluate such medical interviews, thereby alleviating the need for human examiners during training. Yet, real French OSCE annotated transcripts remain extremely scarce, limiting reproducible research and reliable benchmarking. To address these challenges, we investigate the use of LLMs for both generating and evaluating French OSCE dialogues in a low-resource context. We introduce a controlled pipeline that produces synthetic doctor-patient interview transcripts guided by scenario-specific evaluation criteria, combining ideal and perturbed performances to simulate varying student skill levels. The resulting dialogues are automatically silver-labeled through an LLM-assisted framework supporting adjustable evaluation strictness. Benchmarking multiple open-source and proprietary LLMs shows that mid-size models (≤32B parameters) achieve accuracies comparable to GPT-4o (~90%) on synthetic data, highlighting the feasibility of locally deployable, privacy-preserving evaluation systems for medical education.