Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
We present Ko-MuSR, the first benchmark to comprehensively evaluate multistep, soft reasoning in long Korean narratives while minimizing data contamination. Built following MuSR, Ko-MuSR features fully Korean narratives, reasoning chains, and multiple-choice questions verified by human annotators for logical consistency and answerability. Evaluations of four large language models -- two multilingual and two Korean-specialized -- show that multilingual models outperform Korean-focused ones even in Korean reasoning tasks, indicating cross-lingual generalization of reasoning ability. Carefully designed prompting strategies, which combine few-shot examples, reasoning traces, and task-specific hints, further boost accuracy, approaching human-level performance. Ko-MuSR offers a solid foundation for advancing Korean NLP by enabling systematic evaluation of long-context reasoning and prompting strategies.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 16 of its 40 references that are in the map. First posted on arXiv 28 Oct 2025.
Builds on
16 of 40 references.
Closest ancestors · 3
Same direction, newest first
- 2025EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes15 Jul
- 2025HyperCLOVA X THINK Technical Report27 Jun
- 2025Qwen3 Technical Report14 May
Other references · 13
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2023GPT-4 Technical Reportcited by 18,132
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2024OpenAI o1 System Cardcited by 1,220
- 2019Cross-lingual Language Model Pretrainingcited by 1,088
- 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
- 2024Small Language Models: Survey, Measurements, and Insightscited by 109
- 2025Magistralcited by 57
- 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 52
- 2024EXAONE 3.5: Series of Large Language Models for Real-world Use Casescited by 50
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Multi-Step Reasoning in Korean and the Emergent Mirage
- 2025KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs
- 2025Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought
- 2025KORMo: Korean Open Reasoning Model for Everyone
- 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models
- 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning
- 2025Ko-PIQA: A Korean Physical Commonsense Reasoning Dataset with Cultural Context
- 2025Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition