# KOBEST: Korean Balanced Evaluation of Significant Tasks

arXiv: https://arxiv.org/abs/2204.04541 · HTML: https://mlsanity.com/paper/2204.04541/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 45 references that are in the map.
- First posted on arXiv 9 Apr 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Annotation Artifacts in Natural Language Inference Data](https://mlsanity.com/paper/1803.02324/) (2018)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA](https://mlsanity.com/paper/2109.05014/) (2021)
- [CLUE: A Chinese Language Understanding Evaluation Benchmark](https://mlsanity.com/paper/2004.05986/) (2020)
- [Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly](https://mlsanity.com/paper/1911.03343/) (2019)
- [Probing Neural Network Comprehension of Natural Language Arguments](https://mlsanity.com/paper/1907.07355/) (2019)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://mlsanity.com/paper/1912.05372/) (2019)
- [Understanding and Improving Information Transfer in Multi-Task Learning](https://mlsanity.com/paper/2005.00944/) (2020)
- [KLUE: Korean Language Understanding Evaluation](https://mlsanity.com/paper/2105.09680/) (2021)
- [IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding](https://mlsanity.com/paper/2009.05387/) (2020)
- [Understanding by Understanding Not: Modeling Negation in Language Models](https://mlsanity.com/paper/2105.03519/) (2021)
- [The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants](https://mlsanity.com/paper/1708.01425/) (2017)
- [KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding](https://mlsanity.com/paper/2004.03289/) (2020)

## Built on by (most cited first, 30 of 37)

- [KMMLU: Measuring Massive Multitask Language Understanding in Korean](https://mlsanity.com/paper/2402.11548/) (2024)
- [CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean](https://mlsanity.com/paper/2403.06412/) (2024)
- [HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models](https://mlsanity.com/paper/2309.02706/) (2023)
- [LLM-as-a-Judge & Reward Model: What They Can and Cannot Do](https://mlsanity.com/paper/2409.11239/) (2024)
- [Code-Switching Curriculum Learning for Multilingual Transfer in LLMs](https://mlsanity.com/paper/2411.02460/) (2024)
- [A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models](https://mlsanity.com/paper/2306.02254/) (2023)
- [Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models](https://mlsanity.com/paper/2402.14714/) (2024)
- [Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean](https://mlsanity.com/paper/2403.10882/) (2024)
- [Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models](https://mlsanity.com/paper/2504.01001/) (2025)
- [This is the way: designing and compiling LEPISZCZE, a comprehensive NLP benchmark for Polish](https://mlsanity.com/paper/2211.13112/) (2022)
- [KoBALT: Korean Benchmark For Advanced Linguistic Tasks](https://mlsanity.com/paper/2505.16125/) (2025)
- [CABS: Conflict-Aware and Balanced Sparsification for Enhancing Model Merging](https://mlsanity.com/paper/2503.01874/) (2025)
- [Improving Language Models Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionary](https://mlsanity.com/paper/2310.15541/) (2023)
- [Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models](https://mlsanity.com/paper/2410.08731/) (2024)
- [Trillion 7B Technical Report](https://mlsanity.com/paper/2504.15431/) (2025)
- [Merging in a Bottle: Differentiable Adaptive Merging (DAM) and the Path from Averaging to Automation](https://mlsanity.com/paper/2410.08371/) (2024)
- [Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning](https://mlsanity.com/paper/2411.17679/) (2024)
- [K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts](https://mlsanity.com/paper/2606.02404/) (2026)
- [KoDialogBench: Evaluating Conversational Understanding of Language Models with Korean Dialogue Benchmark](https://mlsanity.com/paper/2402.17377/) (2024)
- [Less Is More: Reducing Token Counts Without Compromising Performance](https://mlsanity.com/paper/2506.15138/) (2025)
- [SLM-Based Agentic AI with P-C-G: Optimized for Korean Tool Use](https://mlsanity.com/paper/2509.19369/) (2025)
- [Solar Open Technical Report](https://mlsanity.com/paper/2601.07022/) (2026)
- [ixi-GEN: Efficient Industrial sLLMs through Domain Adaptive Continual Pretraining](https://mlsanity.com/paper/2507.06795/) (2025)
- [Pushing the Boundaries of Multiple Choice Evaluation to One Hundred Options](https://mlsanity.com/paper/2604.14634/) (2026)
- [KIT-19: A Comprehensive Korean Instruction Toolkit on 19 Tasks for Fine-Tuning Korean Large Language Models](https://mlsanity.com/paper/2403.16444/) (2024)
- [Making Sense of Korean Sentences: A Comprehensive Evaluation of LLMs through KoSEnd Dataset](https://mlsanity.com/paper/2507.03378/) (2025)
- [HanjaBridge: Resolving Semantic Ambiguity in Korean LLMs via Hanja-Augmented Pre-Training](https://mlsanity.com/paper/2507.10920/) (2025)
- [Beyond Line-Level Filtering for the Pretraining Corpora of LLMs](https://mlsanity.com/paper/2510.24139/) (2025)
- [SCALE: Upscaled Continual Learning of Large Language Models](https://mlsanity.com/paper/2511.03270/) (2025)
- [ELO: Efficient Layer-Specific Optimization for Continual Pretraining of Multilingual LLMs](https://mlsanity.com/paper/2601.03648/) (2026)

## Abstract (arXiv)

A well-formulated benchmark plays a critical role in spurring advancements in the natural language processing (NLP) field, as it allows objective and precise evaluation of diverse models. As modern language models (LMs) have become more elaborate and sophisticated, more difficult benchmarks that require linguistic knowledge and reasoning have been proposed. However, most of these benchmarks only support English, and great effort is necessary to construct benchmarks for other low resource languages. To this end, we propose a new benchmark named Korean balanced evaluation of significant tasks (KoBEST), which consists of five Korean-language downstream tasks. Professional Korean linguists designed the tasks that require advanced Korean linguistic knowledge. Moreover, our data is purely annotated by humans and thoroughly reviewed to guarantee high data quality. We also provide baseline models and human performance results. Our dataset is available on the Huggingface.