# MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes

arXiv: https://arxiv.org/abs/2412.19260 · HTML: https://mlsanity.com/paper/2412.19260/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 30 references that are in the map.
- First posted on arXiv 26 Dec 2024.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [ScispaCy: Fast and Robust Models for Biomedical Natural Language Processing](https://mlsanity.com/paper/1902.07669/) (2019)
- [CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge](https://mlsanity.com/paper/2109.01653/) (2021)
- [Diagnostic Reasoning Prompts Reveal the Potential for Large Language Model Interpretability in Medicine](https://mlsanity.com/paper/2308.06834/) (2023)
- [SemEval-2020 Task 4: Commonsense Validation and Explanation](https://mlsanity.com/paper/2007.00236/) (2020)
- [Consistency Analysis of ChatGPT](https://mlsanity.com/paper/2303.06273/) (2023)
- [An Investigation of Evaluation Metrics for Automated Medical Note Generation](https://mlsanity.com/paper/2305.17364/) (2023)

## Built on by (most cited first, 30 of 80)

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data](https://mlsanity.com/paper/2510.02410/) (2025)
- [MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluation](https://mlsanity.com/paper/2502.12468/) (2025)
- [Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency](https://mlsanity.com/paper/2506.17209/) (2025)
- [HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats](https://mlsanity.com/paper/2604.27470/) (2026)
- [Revisiting Uncertainty Estimation and Calibration of Large Language Models](https://mlsanity.com/paper/2505.23854/) (2025)
- [A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment](https://mlsanity.com/paper/2505.10717/) (2025)
- [TableMaster: A Recipe to Advance Table Understanding with Language Models](https://mlsanity.com/paper/2501.19378/) (2025)
- [RL of Thoughts: Navigating LLM Reasoning with Inference-time Reinforcement Learning](https://mlsanity.com/paper/2505.14140/) (2025)
- [Auto-Formulating Dynamic Programming Problems with Large Language Models](https://mlsanity.com/paper/2507.11737/) (2025)
- [Enhancing Large Language Models for Hardware Verification: A Novel SystemVerilog Assertion Dataset](https://mlsanity.com/paper/2503.08923/) (2025)
- [Medical large language models are easily distracted](https://mlsanity.com/paper/2504.01201/) (2025)
- [TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law](https://mlsanity.com/paper/2507.21134/) (2025)
- [DAT: Dynamic Alpha Tuning for Hybrid Retrieval in Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.23013/) (2025)
- [LLM-Enhanced Black-Litterman Portfolio Optimization](https://mlsanity.com/paper/2504.14345/) (2025)
- [TeXpert: A Multi-Level Benchmark for Evaluating LaTeX Code Generation by LLMs](https://mlsanity.com/paper/2506.16990/) (2025)
- [Empowering Healthcare Practitioners with Language Models: Structuring Speech Transcripts in Two Real-World Clinical Applications](https://mlsanity.com/paper/2507.05517/) (2025)
- [Bridging the Gap in Ophthalmic AI: MM-Retinal-Reason Dataset and OphthaReason Model toward Dynamic Multimodal Reasoning](https://mlsanity.com/paper/2508.16129/) (2025)
- [E2Edev: Benchmarking Large Language Models in End-to-End Software Development Task](https://mlsanity.com/paper/2510.14509/) (2025)
- [Think Before You Retrieve: Learning Test-Time Adaptive Search with Small Language Models](https://mlsanity.com/paper/2511.07581/) (2025)
- [SocialNav-MoE: A Mixture-of-Experts Vision Language Model for Socially Compliant Navigation with Reinforcement Fine-Tuning](https://mlsanity.com/paper/2512.14757/) (2025)
- [Reproducibility Study of Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation](https://mlsanity.com/paper/2502.16242/) (2025)
- [Policy Frameworks for Transparent Chain-of-Thought Reasoning in Large Language Models](https://mlsanity.com/paper/2503.14521/) (2025)
- [Reasoning LLMs for User-Aware Multimodal Conversational Agents](https://mlsanity.com/paper/2504.01700/) (2025)
- [Empowering Tabular Data Preparation with Language Models: Why and How?](https://mlsanity.com/paper/2508.01556/) (2025)
- [Ontology Learning with LLMs: A Benchmark Study on Axiom Identification](https://mlsanity.com/paper/2512.05594/) (2025)
- [Exploring Multimodal Perception in Large Language Models Through Perceptual Strength Ratings](https://mlsanity.com/paper/2503.06980/) (2025)
- [MedVAL: Toward Expert-Level Medical Text Validation with Language Models](https://mlsanity.com/paper/2507.03152/) (2025)
- [Structured Prompts Improve Evaluation of Language Models](https://mlsanity.com/paper/2511.20836/) (2025)

## Abstract (arXiv)

Several studies showed that Large Language Models (LLMs) can answer medical questions correctly, even outperforming the average human score in some medical exams. However, to our knowledge, no study has been conducted to assess the ability of language models to validate existing or generated medical text for correctness and consistency. In this paper, we introduce MEDEC (https://github.com/abachaa/MEDEC), the first publicly available benchmark for medical error detection and correction in clinical notes, covering five types of errors (Diagnosis, Management, Treatment, Pharmacotherapy, and Causal Organism). MEDEC consists of 3,848 clinical texts, including 488 clinical notes from three US hospital systems that were not previously seen by any LLM. The dataset has been used for the MEDIQA-CORR shared task to evaluate seventeen participating systems [Ben Abacha et al., 2024]. In this paper, we describe the data creation methods and we evaluate recent LLMs (e.g., o1-preview, GPT-4, Claude 3.5 Sonnet, and Gemini 2.0 Flash) for the tasks of detecting and correcting medical errors requiring both medical knowledge and reasoning capabilities. We also conducted a comparative study where two medical doctors performed the same task on the MEDEC test set. The results showed that MEDEC is a sufficiently challenging benchmark to assess the ability of models to validate existing or generated notes and to correct medical errors. We also found that although recent LLMs have a good performance in error detection and correction, they are still outperformed by medical doctors in these tasks. We discuss the potential factors behind this gap, the insights from our experiments, the limitations of current evaluation metrics, and share potential pointers for future research.