# From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation

arXiv: https://arxiv.org/abs/2507.08924 · HTML: https://mlsanity.com/paper/2507.08924/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 47 references that are in the map.
- First posted on arXiv 11 Jul 2025.

## Builds on (most cited first, 27 of 27)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation](https://mlsanity.com/paper/2412.03304/) (2024)
- [FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language](https://mlsanity.com/paper/2506.20920/) (2025)
- [Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier](https://mlsanity.com/paper/2412.04261/) (2024)
- [CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean](https://mlsanity.com/paper/2403.06412/) (2024)
- [PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts](https://mlsanity.com/paper/2504.18428/) (2025)
- [EXAONE 3.5: Series of Large Language Models for Real-world Use Cases](https://mlsanity.com/paper/2412.04862/) (2024)
- [HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models](https://mlsanity.com/paper/2309.02706/) (2023)
- [mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Models](https://mlsanity.com/paper/2406.02301/) (2024)
- [P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMs](https://mlsanity.com/paper/2411.09116/) (2024)
- [Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark](https://mlsanity.com/paper/2405.20574/) (2024)
- [KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations](https://mlsanity.com/paper/2403.01469/) (2024)
- [EXAONE Deep: Reasoning Enhanced Language Models](https://mlsanity.com/paper/2503.12524/) (2025)
- [Spanish and LLM Benchmarks: is MMLU Lost in Translation?](https://mlsanity.com/paper/2406.17789/) (2024)
- [Kanana: Compute-efficient Bilingual Language Models](https://mlsanity.com/paper/2502.18934/) (2025)
- [SQuARe: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created Through Human-Machine Collaboration](https://mlsanity.com/paper/2305.17696/) (2023)
- [Open Ko-LLM Leaderboard2: Bridging Foundational and Practical Evaluation for Korean LLMs](https://mlsanity.com/paper/2410.12445/) (2024)

## Built on by (most cited first, 6 of 6)

- [Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought](https://mlsanity.com/paper/2510.04230/) (2025)
- [KORMo: Korean Open Reasoning Model for Everyone](https://mlsanity.com/paper/2510.09426/) (2025)
- [K-EXAONE Technical Report](https://mlsanity.com/paper/2601.01739/) (2026)
- [KAIO: A Collection of More Challenging Korean Questions](https://mlsanity.com/paper/2509.14752/) (2025)
- [K-EXAONE 2.0 Technical Report](https://mlsanity.com/paper/2608.04505/) (2026)
- [A Factorial Ablation of a Speech-to-SFT Pipeline: Differential Effects on Data Quality and Downstream Transfer](https://mlsanity.com/paper/2608.20394/) (2026)

## Abstract (arXiv)

The development of Large Language Models (LLMs) requires robust benchmarks that encompass not only academic domains but also industrial fields to effectively evaluate their applicability in real-world scenarios. In this paper, we introduce two Korean expert-level benchmarks. KMMLU-Redux, reconstructed from the existing KMMLU, consists of questions from the Korean National Technical Qualification exams, with critical errors removed to enhance reliability. KMMLU-Pro is based on Korean National Professional Licensure exams to reflect professional knowledge in Korea. Our experiments demonstrate that these benchmarks comprehensively represent industrial knowledge in Korea. We release our dataset publicly available.