# MedErrBench: A Fine-Grained Multilingual Benchmark for Medical Error Detection and Correction with Clinical Expert Annotations

arXiv: https://arxiv.org/abs/2602.05692 · HTML: https://mlsanity.com/paper/2602.05692/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 47 references that are in the map.
- First posted on arXiv 5 Feb 2026.

## Builds on (most cited first, 21 of 21)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs](https://mlsanity.com/paper/2412.18925/) (2024)
- [When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs](https://mlsanity.com/paper/2406.01297/) (2024)
- [MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes](https://mlsanity.com/paper/2412.19260/) (2024)
- [Evaluating LLMs at Detecting Errors in LLM Responses](https://mlsanity.com/paper/2404.03602/) (2024)
- [Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods](https://mlsanity.com/paper/2305.18156/) (2023)
- [AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotators](https://mlsanity.com/paper/2402.11073/) (2024)
- [GrammarGPT: Exploring Open-Source LLMs for Native Chinese Grammatical Error Correction with Supervised Fine-Tuning](https://mlsanity.com/paper/2307.13923/) (2023)
- [Enhancing Grammatical Error Correction Systems with Explanations](https://mlsanity.com/paper/2305.15676/) (2023)
- [MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks](https://mlsanity.com/paper/2505.03427/) (2025)
- [FactCheck Editor: Multilingual Text Editor with End-to-End fact-checking](https://mlsanity.com/paper/2404.19482/) (2024)
- [Evaluating Prompting Strategies for Grammatical Error Correction Based on Language Proficiency](https://mlsanity.com/paper/2402.15930/) (2024)
- [Chain-of-Though (CoT) prompting strategies for medical error detection and correction](https://mlsanity.com/paper/2406.09103/) (2024)
- [IryoNLP at MEDIQA-CORR 2024: Tackling the Medical Error Detection & Correction Task On the Shoulders of Medical Agents](https://mlsanity.com/paper/2404.15488/) (2024)
- [PromptMind Team at MEDIQA-CORR 2024: Improving Clinical Text Correction with Error Categorization and LLM Ensembles](https://mlsanity.com/paper/2405.08373/) (2024)
- [MediFact at MEDIQA-CORR 2024: Why AI Needs a Human Touch](https://mlsanity.com/paper/2404.17999/) (2024)
- [MedAraBench: Large-Scale Arabic Medical Question Answering Dataset and Benchmark](https://mlsanity.com/paper/2602.01714/) (2026)
- [MedRECT: A Medical Reasoning Benchmark for Error Correction in Clinical Texts](https://mlsanity.com/paper/2511.00421/) (2025)

## Built on by (most cited first, 2 of 2)

- [MedGuards: Multi-Agent System for Reliable Medical Error Detection and Correction](https://mlsanity.com/paper/2606.25651/) (2026)
- [Toward Better Assessment of LLMs' Performance in Clinical Error Detection](https://mlsanity.com/paper/2608.16643/) (2026)

## Abstract (arXiv)

Inaccuracies in existing or generated clinical text may lead to serious adverse consequences, especially if it is a misdiagnosis or incorrect treatment suggestion. With Large Language Models (LLMs) increasingly being used across diverse healthcare applications, comprehensive evaluation through dedicated benchmarks is crucial. However, such datasets remain scarce, especially across diverse languages and contexts. In this paper, we introduce MedErrBench, the first multilingual benchmark for error detection, localization, and correction, developed under the guidance of experienced clinicians. Based on an expanded taxonomy of ten common error types, MedErrBench covers English, Arabic and Chinese, with natural clinical cases annotated and reviewed by domain experts. We assessed the performance of a range of general-purpose, language-specific, and medical-domain language models across all three tasks. Our results reveal notable performance gaps, particularly in non-English settings, highlighting the need for clinically grounded, language-aware systems. By making MedErrBench and our evaluation protocols publicly-available, we aim to advance multilingual clinical NLP to promote safer and more equitable AI-based healthcare globally. The dataset is available in the supplementary material. An anonymized version of the dataset is available at: https://github.com/congboma/MedErrBench.