mlsanityData to 7 Oct 2026

Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark

arXiv 2405.20574 ↗First posted 31 May 2024cs.CL cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

This paper introduces the Open Ko-LLM Leaderboard and the Ko-H5 Benchmark as vital tools for evaluating Large Language Models (LLMs) in Korean. Incorporating private test sets while mirroring the English Open LLM Leaderboard, we establish a robust evaluation framework that has been well integrated in the Korean LLM community. We perform data leakage analysis that shows the benefit of private test sets along with a correlation study within the Ko-H5 benchmark and temporal analyses of the Ko-H5 score. Moreover, we present empirical support for the need to expand beyond set benchmarks. We hope the Open Ko-LLM Leaderboard sets precedent for expanding LLM evaluation to foster more linguistic diversity.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 19 of its 28 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

19 of 28 references.

References · 19

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  10. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  11. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  12. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
7 more
  1. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  2. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  3. 2023Holistic Evaluation of Text-To-Image Modelscited by 160
  4. 2022MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generationcited by 132
  5. 2021Neural Machine Translation for Low-Resource Languages: A Surveycited by 108
  6. 2020Low-resource Languages: A Review of Past Work and Future Challengescited by 90
  7. 2020An Empirical Study of Tokenization Strategies for Various Korean NLP Taskscited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 13; 2026: 5.

23 papers.

Sort:

By area:

  1. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gapcited by 41
  2. 2025KoBALT: Korean Benchmark For Advanced Linguistic Taskscited by 12
  3. 2024Open Ko-LLM Leaderboard2: Bridging Foundational and Practical Evaluation for Korean LLMscited by 6
  4. 2025From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluationcited by 6
  5. 2024Can Code-Switched Texts Activate a Knowledge Switch in LLMs? A Case Study on English-Korean Code-Switchingcited by 5
  6. 2025TLUE: A Tibetan Language Understanding Evaluation Benchmarkcited by 5
  7. 2024Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Modelscited by 4
  8. 2025Multi-Step Reasoning in Korean and the Emergent Miragecited by 4
  9. 2025Revitalizing Saturated Benchmarks: A Weighted Metric Approach for Differentiating Large Language Model Performancecited by 4
  10. 2025Language Surgery in Multilingual Large Language Modelscited by 4
  11. 2026TurkBench: A Benchmark for Evaluating Turkish Large Language Modelscited by 4
  12. 2025SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligencecited by 3
11 more
  1. 2024Understanding LLM Development Through Longitudinal Study: Insights from the Open Ko-LLM Leaderboardcited by 2
  2. 2024Representing the Under-Represented: Cultural and Core Capability Benchmarks for Developing Thai Large Language Modelscited by 2
  3. 2025Evaluating Multimodal Generative AI with Korean Educational Standardscited by 2
  4. 2025KAIO: A Collection of More Challenging Korean Questionscited by 1
  5. 2026K-FinHallu: A Hallucination Detection Benchmark for Multi-Turn RAG in Korean Financecited by 1
  6. 2025KFinEval-Pilot: A Comprehensive Benchmark Suite for Korean Financial Language Understandingcited by 0
  7. 2025Making Sense of Korean Sentences: A Comprehensive Evaluation of LLMs through KoSEnd Datasetcited by 0
  8. 2025KITE: A Benchmark for Evaluating Korean Instruction-Following Abilities in Large Language Modelscited by 0
  9. 2026HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Modelscited by 0
  10. 2026Llamion Technical Reportcited by 0
  11. 2026KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologismscited by 0