# KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations

arXiv: https://arxiv.org/abs/2403.01469 · HTML: https://mlsanity.com/paper/2403.01469/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 21 references that are in the map.
- First posted on arXiv 3 Mar 2024.

## Builds on (most cited first, 15 of 15)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering](https://mlsanity.com/paper/2203.14371/) (2022)
- [Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine](https://mlsanity.com/paper/2311.16452/) (2023)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [MEDITRON-70B: Scaling Medical Pretraining for Large Language Models](https://mlsanity.com/paper/2311.16079/) (2023)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [HEAD-QA: A Healthcare Dataset for Complex Reasoning](https://mlsanity.com/paper/1906.04701/) (2019)
- [KMMLU: Measuring Massive Multitask Language Understanding in Korean](https://mlsanity.com/paper/2402.11548/) (2024)
- [FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domain](https://mlsanity.com/paper/2304.04280/) (2023)
- [GPT-4's assessment of its performance in a USMLE-based case study](https://mlsanity.com/paper/2402.09654/) (2024)

## Built on by (most cited first, 22 of 22)

- [Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers](https://mlsanity.com/paper/2406.14325/) (2024)
- [ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?](https://mlsanity.com/paper/2411.06469/) (2024)
- [Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts](https://mlsanity.com/paper/2410.10626/) (2024)
- [WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation](https://mlsanity.com/paper/2410.12722/) (2024)
- [HyperCLOVA X THINK Technical Report](https://mlsanity.com/paper/2506.22403/) (2025)
- [From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation](https://mlsanity.com/paper/2507.08924/) (2025)
- [Model Fusion through Bayesian Optimization in Language Model Fine-Tuning](https://mlsanity.com/paper/2411.06710/) (2024)
- [Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub](https://mlsanity.com/paper/2506.00482/) (2025)
- [Solar Open 2 Technical Report](https://mlsanity.com/paper/2607.20062/) (2026)
- [MORE-CLEAR: Multimodal Offline Reinforcement learning for Clinical notes Leveraged Enhanced State Representation](https://mlsanity.com/paper/2508.07681/) (2025)
- [Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study](https://mlsanity.com/paper/2509.03972/) (2025)
- [Solar Open Technical Report](https://mlsanity.com/paper/2601.07022/) (2026)
- [Redefining Evaluation Standards: A Unified Framework for Evaluating the Korean Capabilities of Language Models](https://mlsanity.com/paper/2503.22968/) (2025)
- [Format Inertia: A Failure Mechanism of LLMs in Medical Pre-Consultation](https://mlsanity.com/paper/2510.01688/) (2025)
- [Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines](https://mlsanity.com/paper/2601.03627/) (2026)
- [KorMedMCQA-V: A Multimodal Benchmark for Evaluating Vision-Language Models on the Korean Medical Licensing Examination](https://mlsanity.com/paper/2602.13650/) (2026)
- [Towards Democratizing Multilingual Large Language Models For Medicine Through A Two-Stage Instruction Fine-tuning Approach](https://mlsanity.com/paper/2409.05732/) (2024)
- [VM14K: First Vietnamese Medical Benchmark](https://mlsanity.com/paper/2506.01305/) (2025)
- [ControlMed: Adding Reasoning Control to Medical Language Model](https://mlsanity.com/paper/2507.22545/) (2025)
- [JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation](https://mlsanity.com/paper/2605.22080/) (2026)
- [PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoning](https://mlsanity.com/paper/2609.04598/) (2026)
- [A Multi-Agent LLM Framework for Personalized Health Checkup Interpretation and Guidance](https://mlsanity.com/paper/2610.01451/) (2026)

## Abstract (arXiv)

We present KorMedMCQA, the first Korean Medical Multiple-Choice Question Answering benchmark, derived from professional healthcare licensing examinations conducted in Korea between 2012 and 2024. The dataset contains 7,469 questions from examinations for doctor, nurse, pharmacist, and dentist, covering a wide range of medical disciplines. We evaluate the performance of 59 large language models, spanning proprietary and open-source models, multilingual and Korean-specialized models, and those fine-tuned for clinical applications. Our results show that applying Chain of Thought (CoT) reasoning can enhance the model performance by up to 4.5% compared to direct answering approaches. We also investigate whether MedQA, one of the most widely used medical benchmarks derived from the U.S. Medical Licensing Examination, can serve as a reliable proxy for evaluating model performance in other regions-in this case, Korea. Our correlation analysis between model scores on KorMedMCQA and MedQA reveals that these two benchmarks align no better than benchmarks from entirely different domains (e.g., MedQA and MMLU-Pro). This finding underscores the substantial linguistic and clinical differences between Korean and U.S. medical contexts, reinforcing the need for region-specific medical QA benchmarks. To support ongoing research in Korean healthcare AI, we publicly release the KorMedMCQA via Huggingface.