# KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Context

arXiv: https://arxiv.org/abs/2412.07251 · HTML: https://mlsanity.com/paper/2412.07251/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 16 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Builds on (most cited first, 12 of 12)

- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [CMMLU: Measuring massive multitask language understanding in Chinese](https://mlsanity.com/paper/2306.09212/) (2023)
- [What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers](https://mlsanity.com/paper/2109.04650/) (2021)
- [KoBBQ: Korean Bias Benchmark for Question Answering](https://mlsanity.com/paper/2307.16778/) (2023)
- [CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean](https://mlsanity.com/paper/2403.06412/) (2024)
- [HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models](https://mlsanity.com/paper/2309.02706/) (2023)
- [KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge](https://mlsanity.com/paper/2402.13605/) (2024)
- [IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context](https://mlsanity.com/paper/2403.20147/) (2024)
- [CCPM: A Chinese Classical Poetry Matching Dataset](https://mlsanity.com/paper/2106.01979/) (2021)
- [Heron-Bench: A Benchmark for Evaluating Vision Language Models in Japanese](https://mlsanity.com/paper/2404.07824/) (2024)

## Built on by (most cited first, 8 of 8)

- [Memorization or Reasoning? Exploring the Idiom Understanding of LLMs](https://mlsanity.com/paper/2505.16216/) (2025)
- [Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub](https://mlsanity.com/paper/2506.00482/) (2025)
- [MyCulture: Exploring Malaysia's Diverse Culture under Low-Resource Language Constraints](https://mlsanity.com/paper/2508.05429/) (2025)
- [Can Language Models Handle a Non-Gregorian Calendar? The Case of the Japanese wareki](https://mlsanity.com/paper/2509.04432/) (2025)
- [No Shortcuts to Culture: Indonesian Multi-hop Question Answering for Complex Cultural Understanding](https://mlsanity.com/paper/2602.03709/) (2026)
- [Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims](https://mlsanity.com/paper/2606.02493/) (2026)
- [VIVID: A Culturally Grounded Benchmark Exposing the Figurative Language Gap in Vietnamese NLP](https://mlsanity.com/paper/2608.03095/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)

## Abstract (arXiv)

Large language models have exhibited significant enhancements in performance across various tasks. However, the complexity of their evaluation increases as these models generate more fluent and coherent content. Current multilingual benchmarks often use translated English versions, which may incorporate Western cultural biases that do not accurately assess other languages and cultures. To address this research gap, we introduce KULTURE Bench, an evaluation framework specifically designed for Korean culture that features datasets of cultural news, idioms, and poetry. It is designed to assess language models' cultural comprehension and reasoning capabilities at the word, sentence, and paragraph levels. Using the KULTURE Bench, we assessed the capabilities of models trained with different language corpora and analyzed the results comprehensively. The results show that there is still significant room for improvement in the models' understanding of texts related to the deeper aspects of Korean culture.