# Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark

arXiv: https://arxiv.org/abs/2405.20574 · HTML: https://mlsanity.com/paper/2405.20574/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 31 May 2024.

## Builds on (most cited first, 19 of 19)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Don't Make Your LLM an Evaluation Benchmark Cheater](https://mlsanity.com/paper/2311.01964/) (2023)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [Holistic Evaluation of Text-To-Image Models](https://mlsanity.com/paper/2311.04287/) (2023)
- [MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation](https://mlsanity.com/paper/2208.08227/) (2022)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Low-resource Languages: A Review of Past Work and Future Challenges](https://mlsanity.com/paper/2006.07264/) (2020)
- [An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks](https://mlsanity.com/paper/2010.02534/) (2020)

## Built on by (most cited first, 23 of 23)

- [Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap](https://mlsanity.com/paper/2501.02448/) (2025)
- [KoBALT: Korean Benchmark For Advanced Linguistic Tasks](https://mlsanity.com/paper/2505.16125/) (2025)
- [Open Ko-LLM Leaderboard2: Bridging Foundational and Practical Evaluation for Korean LLMs](https://mlsanity.com/paper/2410.12445/) (2024)
- [TLUE: A Tibetan Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2503.12051/) (2025)
- [From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation](https://mlsanity.com/paper/2507.08924/) (2025)
- [Can Code-Switched Texts Activate a Knowledge Switch in LLMs? A Case Study on English-Korean Code-Switching](https://mlsanity.com/paper/2410.18436/) (2024)
- [TurkBench: A Benchmark for Evaluating Turkish Large Language Models](https://mlsanity.com/paper/2601.07020/) (2026)
- [Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models](https://mlsanity.com/paper/2410.08731/) (2024)
- [Multi-Step Reasoning in Korean and the Emergent Mirage](https://mlsanity.com/paper/2501.05712/) (2025)
- [Revitalizing Saturated Benchmarks: A Weighted Metric Approach for Differentiating Large Language Model Performance](https://mlsanity.com/paper/2503.05551/) (2025)
- [Language Surgery in Multilingual Large Language Models](https://mlsanity.com/paper/2506.12450/) (2025)
- [SciEvalKit: An Open-source Evaluation Toolkit for Scientific General Intelligence](https://mlsanity.com/paper/2512.22334/) (2025)
- [Understanding LLM Development Through Longitudinal Study: Insights from the Open Ko-LLM Leaderboard](https://mlsanity.com/paper/2409.03257/) (2024)
- [Representing the Under-Represented: Cultural and Core Capability Benchmarks for Developing Thai Large Language Models](https://mlsanity.com/paper/2410.04795/) (2024)
- [Evaluating Multimodal Generative AI with Korean Educational Standards](https://mlsanity.com/paper/2502.15422/) (2025)
- [KAIO: A Collection of More Challenging Korean Questions](https://mlsanity.com/paper/2509.14752/) (2025)
- [K-FinHallu: A Hallucination Detection Benchmark for Multi-Turn RAG in Korean Finance](https://mlsanity.com/paper/2605.29523/) (2026)
- [KFinEval-Pilot: A Comprehensive Benchmark Suite for Korean Financial Language Understanding](https://mlsanity.com/paper/2504.13216/) (2025)
- [Making Sense of Korean Sentences: A Comprehensive Evaluation of LLMs through KoSEnd Dataset](https://mlsanity.com/paper/2507.03378/) (2025)
- [KITE: A Benchmark for Evaluating Korean Instruction-Following Abilities in Large Language Models](https://mlsanity.com/paper/2510.15558/) (2025)
- [HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models](https://mlsanity.com/paper/2604.19786/) (2026)
- [Llamion Technical Report](https://mlsanity.com/paper/2605.25676/) (2026)
- [KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms](https://mlsanity.com/paper/2609.19916/) (2026)

## Abstract (arXiv)

This paper introduces the Open Ko-LLM Leaderboard and the Ko-H5 Benchmark as vital tools for evaluating Large Language Models (LLMs) in Korean. Incorporating private test sets while mirroring the English Open LLM Leaderboard, we establish a robust evaluation framework that has been well integrated in the Korean LLM community. We perform data leakage analysis that shows the benefit of private test sets along with a correlation study within the Ko-H5 benchmark and temporal analyses of the Ko-H5 score. Moreover, we present empirical support for the need to expand beyond set benchmarks. We hope the Open Ko-LLM Leaderboard sets precedent for expanding LLM evaluation to foster more linguistic diversity.