# Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition

arXiv: https://arxiv.org/abs/2507.04014 · HTML: https://mlsanity.com/paper/2507.04014/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 10 of its 19 references that are in the map.
- First posted on arXiv 5 Jul 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://mlsanity.com/paper/2406.09948/) (2024)
- [CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean](https://mlsanity.com/paper/2403.06412/) (2024)
- [CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge](https://mlsanity.com/paper/2404.06664/) (2024)
- [EXAONE 3.0 7.8B Instruction Tuned Language Model](https://mlsanity.com/paper/2408.03541/) (2024)
- [HyperCLOVA X Technical Report](https://mlsanity.com/paper/2404.01954/) (2024)
- [General Purpose Verification for Chain of Thought Prompting](https://mlsanity.com/paper/2405.00204/) (2024)
- [Evaluating Human Alignment and Model Faithfulness of LLM Rationale](https://mlsanity.com/paper/2407.00219/) (2024)

## Built on by (most cited first, 6 of 6)

- [BengaliMoralBench: A Benchmark for Auditing Moral Reasoning in Large Language Models within Bengali Language and Culture](https://mlsanity.com/paper/2511.03180/) (2025)
- [Unmasking the Factual-Conceptual Gap in Persian Language Models](https://mlsanity.com/paper/2602.17623/) (2026)
- [Defining Cultural Capabilities for AI Evaluation: A Taxonomy Grounded in Intercultural Communication Theory](https://mlsanity.com/paper/2605.15990/) (2026)
- [Toward Culturally Grounded Natural Language Processing](https://mlsanity.com/paper/2603.26013/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)
- [CRISP: Cultural Reward Modeling for Implicit Situated Propriety](https://mlsanity.com/paper/2609.34345/) (2026)

## Abstract (arXiv)

As large language models (LLMs) become key advisors in various domains, their cultural sensitivity and reasoning skills are crucial in multicultural environments. We introduce Nunchi-Bench, a benchmark designed to evaluate LLMs' cultural understanding, with a focus on Korean superstitions. The benchmark consists of 247 questions spanning 31 topics, assessing factual knowledge, culturally appropriate advice, and situational interpretation. We evaluate multilingual LLMs in both Korean and English to analyze their ability to reason about Korean cultural contexts and how language variations affect performance. To systematically assess cultural reasoning, we propose a novel evaluation strategy with customized scoring metrics that capture the extent to which models recognize cultural nuances and respond appropriately. Our findings highlight significant challenges in LLMs' cultural reasoning. While models generally recognize factual information, they struggle to apply it in practical scenarios. Furthermore, explicit cultural framing enhances performance more effectively than relying solely on the language of the prompt. To support further research, we publicly release Nunchi-Bench alongside a leaderboard.