# Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean

arXiv: https://arxiv.org/abs/2510.24150 · HTML: https://mlsanity.com/paper/2510.24150/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 16 of its 40 references that are in the map.
- First posted on arXiv 28 Oct 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [Small Language Models: Survey, Measurements, and Insights](https://mlsanity.com/paper/2409.15790/) (2024)
- [Magistral](https://mlsanity.com/paper/2506.10910/) (2025)
- [CroissantLLM: A Truly Bilingual French-English Language Model](https://mlsanity.com/paper/2402.00786/) (2024)
- [EXAONE 3.5: Series of Large Language Models for Real-world Use Cases](https://mlsanity.com/paper/2412.04862/) (2024)
- [Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap](https://mlsanity.com/paper/2501.02448/) (2025)
- [EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modes](https://mlsanity.com/paper/2507.11407/) (2025)
- [HyperCLOVA X THINK Technical Report](https://mlsanity.com/paper/2506.22403/) (2025)

## Abstract (arXiv)

We present Ko-MuSR, the first benchmark to comprehensively evaluate multistep, soft reasoning in long Korean narratives while minimizing data contamination. Built following MuSR, Ko-MuSR features fully Korean narratives, reasoning chains, and multiple-choice questions verified by human annotators for logical consistency and answerability. Evaluations of four large language models -- two multilingual and two Korean-specialized -- show that multilingual models outperform Korean-focused ones even in Korean reasoning tasks, indicating cross-lingual generalization of reasoning ability. Carefully designed prompting strategies, which combine few-shot examples, reasoning traces, and task-specific hints, further boost accuracy, approaching human-level performance. Ko-MuSR offers a solid foundation for advancing Korean NLP by enabling systematic evaluation of long-context reasoning and prompting strategies.