# KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs

arXiv: https://arxiv.org/abs/2510.18368 · HTML: https://mlsanity.com/paper/2510.18368/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 16 of its 37 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 16 of 16)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [Do LLMs Know When to NOT Answer? Investigating Abstention Abilities of Large Language Models](https://mlsanity.com/paper/2407.16221/) (2024)
- [SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge](https://mlsanity.com/paper/2509.07968/) (2025)
- [Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models](https://mlsanity.com/paper/2411.07140/) (2024)
- [EXAONE Deep: Reasoning Enhanced Language Models](https://mlsanity.com/paper/2503.12524/) (2025)
- [Kanana: Compute-efficient Bilingual Language Models](https://mlsanity.com/paper/2502.18934/) (2025)
- [HyperCLOVA X Technical Report](https://mlsanity.com/paper/2404.01954/) (2024)
- [Language Models' Factuality Depends on the Language of Inquiry](https://mlsanity.com/paper/2502.17955/) (2025)
- [Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?](https://mlsanity.com/paper/2510.06036/) (2025)
- [It is Too Many Options: Pitfalls of Multiple-Choice Questions in Generative AI and Medical Education](https://mlsanity.com/paper/2503.13508/) (2025)
- [HyperCLOVA X THINK Technical Report](https://mlsanity.com/paper/2506.22403/) (2025)
- [Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering](https://mlsanity.com/paper/2505.16591/) (2025)

## Built on by (most cited first, 1 of 1)

- [From National Curricula to Cultural Awareness: Constructing Open-Ended Culture-Specific Question Answering Dataset](https://mlsanity.com/paper/2601.04632/) (2026)

## Abstract (arXiv)

We present Korean SimpleQA (KoSimpleQA), a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cultural knowledge. KoSimpleQA is designed to be challenging yet easy to grade, consisting of 938 short, fact-seeking questions with unambiguous answers. We conduct a comprehensive evaluation across a diverse set of open-source LLMs of varying sizes that support Korean, and find that even the strongest model generates correct answer only 31.6% of the time, underscoring the challenging nature of KoSimpleQA. Notably, performance rankings on KoSimpleQA differ substantially from those on the English SimpleQA, highlighting the unique value of our dataset. Furthermore, we observe that reasoning helps mitigate the cross-lingual knowledge gap in LLMs, which refers to disparities in their ability to manifest knowledge across languages. KoSimpleQA can be found at https://github.com/naver-ai/KoSimpleQA.