mlsanityData to 14 Oct 2026

From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation

arXiv 2507.08924 ↗First posted 11 Jul 2025cs.CL cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

The development of Large Language Models (LLMs) requires robust benchmarks that encompass not only academic domains but also industrial fields to effectively evaluate their applicability in real-world scenarios. In this paper, we introduce two Korean expert-level benchmarks. KMMLU-Redux, reconstructed from the existing KMMLU, consists of questions from the Korean National Technical Qualification exams, with critical errors removed to enhance reliability. KMMLU-Pro is based on Korean National Professional Licensure exams to reflect professional knowledge in Korea. Our experiments demonstrate that these benchmarks comprehensively represent industrial knowledge in Korea. We release our dataset publicly available.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 27 of its 47 references that are in the map. First posted on arXiv 11 Jul 2025.

Builds on

27 of 47 references.

References · 27

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  11. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  12. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
15 more
  1. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  2. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
  3. 2024CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Koreancited by 66
  4. 2025PolyMath: Evaluating Mathematical Reasoning in Multilingual Contextscited by 56
  5. 2024EXAONE 3.5: Series of Large Language Models for Real-world Use Casescited by 53
  6. 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Modelscited by 47
  7. 2024mCoT: Multilingual Instruction Tuning for Reasoning Consistency in Language Modelscited by 33
  8. 2024P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMscited by 29
  9. 2024Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmarkcited by 23
  10. 2024KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinationscited by 22
  11. 2025EXAONE Deep: Reasoning Enhanced Language Modelscited by 21
  12. 2024Spanish and LLM Benchmarks: is MMLU Lost in Translation?cited by 20
  13. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 20
  14. 2023SQuARe: A Large-Scale Dataset of Sensitive Questions and Acceptable Responses Created Through Human-Machine Collaborationcited by 13
  15. 2024Open Ko-LLM Leaderboard2: Bridging Foundational and Practical Evaluation for Korean LLMscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thoughtcited by 10
  2. 2025KORMo: Korean Open Reasoning Model for Everyonecited by 4
  3. 2026K-EXAONE Technical Reportcited by 2
  4. 2025KAIO: A Collection of More Challenging Korean Questionscited by 1
  5. 2026K-EXAONE 2.0 Technical Reportcited by 0
  6. 2026A Factorial Ablation of a Speech-to-SFT Pipeline: Differential Effects on Data Quality and Downstream Transfercited by 0