mlsanityData to 7 Oct 2026

KMMLU: Measuring Massive Multitask Language Understanding in Korean

arXiv 2402.11548 ↗First posted 18 Feb 2024cs.CLcited by 91

Not on the map. Why?

Abstract (arXiv)

We propose KMMLU, a new Korean benchmark with 35,030 expert-level multiple-choice questions across 45 subjects ranging from humanities to STEM. While prior Korean benchmarks are translated from existing English benchmarks, KMMLU is collected from original Korean exams, capturing linguistic and cultural aspects of the Korean language. We test 27 public and proprietary LLMs and observe the best public model to score 50.5%, leaving significant room for improvement. This model was primarily trained for English and Chinese, not Korean. Current LLMs tailored to Korean, such as Polyglot-Ko, perform far worse. Surprisingly, even the most capable proprietary LLMs, e.g., GPT-4 and HyperCLOVA X do not exceed 60%. This suggests that further work is needed to improve LLMs for Korean, and we believe KMMLU offers the appropriate tool to track this progress. We make our dataset publicly available on the Hugging Face Hub and integrate the benchmark into EleutherAI's Language Model Evaluation Harness.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 44 of its 75 references that are in the map. First posted on arXiv 18 Feb 2024.

Builds on

44 of 75 references.

References · 44

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
32 more
  1. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  13. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  14. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  15. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  16. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  17. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  18. 2021KLUE: Korean Language Understanding Evaluationcited by 113
  19. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 94
  20. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  21. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  22. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  23. 2023KoBBQ: Korean Bias Benchmark for Question Answeringcited by 71
  24. 2019Generalized Data Augmentation for Low-Resource Translationcited by 53
  25. 2023Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLUcited by 53
  26. 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Modelscited by 47
  27. 2019KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehensioncited by 43
  28. 2020BEEP! Korean Corpus of Online News Comments for Toxic Speech Detectioncited by 39
  29. 2020KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understandingcited by 37
  30. 2022KOBEST: Korean Balanced Evaluation of Significant Taskscited by 37
  31. 2023A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Modelscited by 22
  32. 2022FRMT: A Benchmark for Few-Shot Region-Aware Machine Translationcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 22; 2025: 39; 2026: 30.

91 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  2. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  3. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  4. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  5. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluationcited by 73
  6. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  7. 2024EXAONE 3.5: Series of Large Language Models for Real-world Use Casescited by 50
  8. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 47
  9. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 41
  10. 2025Culture is Not Trivia: Sociocultural Theory for Cultural NLPcited by 40
  11. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 34
  12. 2025Linguistic Generalizability of Test-Time Scaling in Mathematical Reasoningcited by 33
38 more
  1. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  2. 2024CaLMQA: Exploring culturally specific long-form question answering across 23 languagescited by 31
  3. 2025The Bitter Lesson Learned from 2,000+ Multilingual Benchmarkscited by 31
  4. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  5. 2024Code-Switching Curriculum Learning for Multilingual Transfer in LLMscited by 26
  6. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  7. 2024BertaQA: How Much Do Language Models Know About Local Culture?cited by 23
  8. 2024KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinationscited by 22
  9. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 19
  10. 2024HyperCLOVA X Technical Reportcited by 18
  11. 2024Navigating the Path of Writing: Outline-guided Text Generation with Large Language Modelscited by 15
  12. 2024ProverbEval: Exploring LLM Evaluation Challenges for Low-resource Language Understandingcited by 15
  13. 2025Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Modelscited by 14
  14. 2024BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanismcited by 12
  15. 2024SUTRA: Scalable Multilingual Language Model Architecturecited by 11
  16. 2025Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thoughtcited by 10
  17. 2024MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Modelscited by 9
  18. 2024Pragmatic Competence Evaluation of Large Language Models for the Korean Languagecited by 8
  19. 2025HKCanto-Eval: A Benchmark for Evaluating Cantonese Language Understanding and Cultural Comprehension in LLMscited by 8
  20. 2025LLMzSzL: a comprehensive LLM benchmark for Polishcited by 7
  21. 2025KoBLEX: Open Legal Question Answering with Multi-hop Reasoningcited by 7
  22. 2025BnMMLU: Measuring Massive Multitask Language Understanding in Bengalicited by 6
  23. 2025SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhalacited by 6
  24. 2025Multi-Step Reasoning in Korean and the Emergent Miragecited by 4
  25. 2025Trillion 7B Technical Reportcited by 4
  26. 2025IRLBench: A Multi-modal, Culturally Grounded, Parallel Irish-English Benchmark for Open-Ended LLM Reasoning Evaluationcited by 4
  27. 2025Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answeringcited by 3
  28. 2025BenchHub: A Unified Benchmark Suite for Holistic and Customizable LLM Evaluationcited by 3
  29. 2025Securing Multi-Agent Systems Against Corruptions via Node Contribution Backpropagationcited by 3
  30. 2025SGuard-v1: Safety Guardrail for Large Language Modelscited by 3
  31. 2026Mi:dm 2.0 Korea-centric Bilingual Language Modelscited by 3
  32. 2026BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarkscited by 3
  33. 2026Apples to Apples? Towards Comparable Crosslingual Language Model Evaluationcited by 3
  34. 2025Evaluating Multimodal Generative AI with Korean Educational Standardscited by 2
  35. 2025LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriamacited by 2
  36. 2025Smoothie-Qwen: Post-Hoc Smoothing to Reduce Language Bias in Multilingual LLMscited by 2
  37. 2025ParamBench: A Graduate-Level Benchmark for Evaluating LLM Understanding on Indic Subjectscited by 2
  38. 2025Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Studycited by 2

50 most cited of 91 All 91 (JSON)