# CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean

arXiv: https://arxiv.org/abs/2403.06412 · HTML: https://mlsanity.com/paper/2403.06412/

## Key facts

- Cited by 66 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 45 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 30 of 31)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://mlsanity.com/paper/2304.06364/) (2023)
- [C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models](https://mlsanity.com/paper/2305.08322/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning](https://mlsanity.com/paper/1909.00277/) (2019)
- [PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification](https://mlsanity.com/paper/1908.11828/) (2019)
- [XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation](https://mlsanity.com/paper/2004.01401/) (2020)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models](https://mlsanity.com/paper/2102.02503/) (2021)
- [KLUE: Korean Language Understanding Evaluation](https://mlsanity.com/paper/2105.09680/) (2021)
- [Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages](https://mlsanity.com/paper/2212.05409/) (2022)
- [Neural Question Generation from Text: A Preliminary Study](https://mlsanity.com/paper/1704.01792/) (2017)
- [KoBBQ: Korean Bias Benchmark for Question Answering](https://mlsanity.com/paper/2307.16778/) (2023)
- [HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models](https://mlsanity.com/paper/2309.02706/) (2023)
- [KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehension](https://mlsanity.com/paper/1909.07005/) (2019)
- [SERENGETI: Massively Multilingual Language Models for Africa](https://mlsanity.com/paper/2212.10785/) (2022)
- [BEEP! Korean Corpus of Online News Comments for Toxic Speech Detection](https://mlsanity.com/paper/2005.12503/) (2020)
- [KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding](https://mlsanity.com/paper/2004.03289/) (2020)
- [KOBEST: Korean Balanced Evaluation of Significant Tasks](https://mlsanity.com/paper/2204.04541/) (2022)
- [A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models](https://mlsanity.com/paper/2306.02254/) (2023)

## Built on by (most cited first, 30 of 66)

- [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://mlsanity.com/paper/2406.09948/) (2024)
- [CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies](https://mlsanity.com/paper/2404.15238/) (2024)
- [Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap](https://mlsanity.com/paper/2501.02448/) (2025)
- [Culture is Not Trivia: Sociocultural Theory for Cultural NLP](https://mlsanity.com/paper/2502.12057/) (2025)
- [CaLMQA: Exploring culturally specific long-form question answering across 23 languages](https://mlsanity.com/paper/2406.17761/) (2024)
- [Code-Switching Curriculum Learning for Multilingual Transfer in LLMs](https://mlsanity.com/paper/2411.02460/) (2024)
- [Extrinsic Evaluation of Cultural Competence in Large Language Models](https://mlsanity.com/paper/2406.11565/) (2024)
- [Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaboration](https://mlsanity.com/paper/2406.16469/) (2024)
- [HyperCLOVA X Technical Report](https://mlsanity.com/paper/2404.01954/) (2024)
- [KoBALT: Korean Benchmark For Advanced Linguistic Tasks](https://mlsanity.com/paper/2505.16125/) (2025)
- [mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans](https://mlsanity.com/paper/2406.04215/) (2024)
- [Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought](https://mlsanity.com/paper/2510.04230/) (2025)
- [MultiHoax: A Dataset of Multi-hop False-Premise Questions](https://mlsanity.com/paper/2506.00264/) (2025)
- [KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Context](https://mlsanity.com/paper/2412.07251/) (2024)
- [PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian](https://mlsanity.com/paper/2502.07459/) (2025)
- [SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabia](https://mlsanity.com/paper/2503.17485/) (2025)
- [CAReDiO: Cultural Alignment via Representativeness and Distinctiveness Guided Data Optimization](https://mlsanity.com/paper/2504.08820/) (2025)
- [Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition](https://mlsanity.com/paper/2507.04014/) (2025)
- [From KMMLU-Redux to KMMLU-Pro: A Professional Korean Benchmark Suite for LLM Evaluation](https://mlsanity.com/paper/2507.08924/) (2025)
- [Can Code-Switched Texts Activate a Knowledge Switch in LLMs? A Case Study on English-Korean Code-Switching](https://mlsanity.com/paper/2410.18436/) (2024)
- [MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation](https://mlsanity.com/paper/2510.08608/) (2025)
- [Self-Explaining Hate Speech Detection with Moral Rationales](https://mlsanity.com/paper/2601.03481/) (2026)
- [Mi:dm 2.0 Korea-centric Bilingual Language Models](https://mlsanity.com/paper/2601.09066/) (2026)
- [Constella: Supporting Storywriters' Interconnected Character Creation through LLM-based Multi-Agents](https://mlsanity.com/paper/2507.05820/) (2025)
- [KORMo: Korean Open Reasoning Model for Everyone](https://mlsanity.com/paper/2510.09426/) (2025)
- [Can LLMs Grasp Implicit Cultural Values? Benchmarking LLMs' Cultural Intelligence with CQ-Bench](https://mlsanity.com/paper/2504.01127/) (2025)
- [BLUCK: A Benchmark Dataset for Bengali Linguistic Understanding and Cultural Knowledge](https://mlsanity.com/paper/2505.21092/) (2025)
- [When Language Shapes Thought: Cross-Lingual Transfer of Factual Knowledge in Question Answering](https://mlsanity.com/paper/2505.24409/) (2025)
- [Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub](https://mlsanity.com/paper/2506.00482/) (2025)
- [MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs](https://mlsanity.com/paper/2507.09701/) (2025)

## Abstract (arXiv)

Despite the rapid development of large language models (LLMs) for the Korean language, there remains an obvious lack of benchmark datasets that test the requisite Korean cultural and linguistic knowledge. Because many existing Korean benchmark datasets are derived from the English counterparts through translation, they often overlook the different cultural contexts. For the few benchmark datasets that are sourced from Korean data capturing cultural knowledge, only narrow tasks such as bias and hate speech detection are offered. To address this gap, we introduce a benchmark of Cultural and Linguistic Intelligence in Korean (CLIcK), a dataset comprising 1,995 QA pairs. CLIcK sources its data from official Korean exams and textbooks, partitioning the questions into eleven categories under the two main categories of language and culture. For each instance in CLIcK, we provide fine-grained annotation of which cultural and linguistic knowledge is required to answer the question correctly. Using CLIcK, we test 13 language models to assess their performance. Our evaluation uncovers insights into their performances across the categories, as well as the diverse factors affecting their comprehension. CLIcK offers the first large-scale comprehensive Korean-centric analysis of LLMs' proficiency in Korean culture and language.