# KoBALT: Korean Benchmark For Advanced Linguistic Tasks

arXiv: https://arxiv.org/abs/2505.16125 · HTML: https://mlsanity.com/paper/2505.16125/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 16 of 16)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean](https://mlsanity.com/paper/2403.06412/) (2024)
- [HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models](https://mlsanity.com/paper/2309.02706/) (2023)
- [KOBEST: Korean Balanced Evaluation of Significant Tasks](https://mlsanity.com/paper/2204.04541/) (2022)
- [LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages](https://mlsanity.com/paper/2406.06196/) (2024)
- [Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark](https://mlsanity.com/paper/2405.20574/) (2024)
- [Dissecting Human and LLM Preferences](https://mlsanity.com/paper/2402.11296/) (2024)
- [Linguini: A benchmark for language-agnostic linguistic reasoning](https://mlsanity.com/paper/2409.12126/) (2024)
- [MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models](https://mlsanity.com/paper/2406.07736/) (2024)
- [PhonologyBench: Evaluating Phonological Skills of Large Language Models](https://mlsanity.com/paper/2404.02456/) (2024)
- [Uncovering Factor Level Preferences to Improve Human-Model Alignment](https://mlsanity.com/paper/2410.06965/) (2024)

## Built on by (most cited first, 13 of 13)

- [Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought](https://mlsanity.com/paper/2510.04230/) (2025)
- [HyperCLOVA X THINK Technical Report](https://mlsanity.com/paper/2506.22403/) (2025)
- [KORMo: Korean Open Reasoning Model for Everyone](https://mlsanity.com/paper/2510.09426/) (2025)
- [Solar Open Technical Report](https://mlsanity.com/paper/2601.07022/) (2026)
- [BanglaSocialBench: A Benchmark for Evaluating Sociopragmatic and Cultural Alignment of LLMs in Bangladeshi Social Interaction](https://mlsanity.com/paper/2603.15949/) (2026)
- [EXAONE 4.5 Technical Report](https://mlsanity.com/paper/2604.08644/) (2026)
- [Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options](https://mlsanity.com/paper/2604.14634/) (2026)
- [HanjaBridge: Resolving Semantic Ambiguity in Korean LLMs via Hanja-Augmented Pre-Training](https://mlsanity.com/paper/2507.10920/) (2025)
- [MATA: Mindful Assessment of the Telugu Abilities of Large Language Models](https://mlsanity.com/paper/2508.13526/) (2025)
- [HyperCLOVA X 32B Think](https://mlsanity.com/paper/2601.03286/) (2026)
- [Mi:dm K 2.5 Pro](https://mlsanity.com/paper/2603.18788/) (2026)
- [Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese](https://mlsanity.com/paper/2606.07300/) (2026)
- [BLiMPPI: Benchmark of Linguistic Minimal Pairs for Polarity Items](https://mlsanity.com/paper/2610.13607/) (2026)

## Abstract (arXiv)

We introduce KoBALT (Korean Benchmark for Advanced Linguistic Tasks), a comprehensive linguistically-motivated benchmark comprising 700 multiple-choice questions spanning 24 phenomena across five linguistic domains: syntax, semantics, pragmatics, phonetics/phonology, and morphology. KoBALT is designed to advance the evaluation of large language models (LLMs) in Korean, a morphologically rich language, by addressing the limitations of conventional benchmarks that often lack linguistic depth and typological grounding. It introduces a suite of expert-curated, linguistically motivated questions with minimal n-gram overlap with standard Korean corpora, substantially mitigating the risk of data contamination and allowing a more robust assessment of true language understanding. Our evaluation of 20 contemporary LLMs reveals significant performance disparities, with the highest-performing model achieving 61% general accuracy but showing substantial variation across linguistic domains - from stronger performance in semantics (66%) to considerable weaknesses in phonology (31%) and morphology (36%). Through human preference evaluation with 95 annotators, we demonstrate a strong correlation between KoBALT scores and human judgments, validating our benchmark's effectiveness as a discriminative measure of Korean language understanding. KoBALT addresses critical gaps in linguistic evaluation for typologically diverse languages and provides a robust framework for assessing genuine linguistic competence in Korean language models.