mlsanityData to 7 Oct 2026

NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gap

arXiv 2608.04397 ↗First posted 5 Aug 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce NOLLI, a procedurally generated English-Korean puzzle benchmark designed to diagnose where Korean performance gaps arise. It comprises 15 puzzle types (25 tasks; 7,500 items), with every instance seed-regenerable, verified to have a unique solution, and scored deterministically. Rather than equating harder with bigger, we calibrate difficulty behaviorally, tuning each generator until a fixed reference model lands in target accuracy bands. Its three-level design combines matched direct translations, script adaptations over Hangul jamo (sub-syllabic letters), and Korean-only tasks grounded in Korean culture or orthography. We evaluate 15 frontier, open-weight, and Korean-developed models; among the 12 above a 3% overall-accuracy floor, matched English-Korean accuracy is statistically equivalent within a +/- 10 pp margin (TOST), suggesting little cost from presentation language alone. Writing-system-intensive tasks show sharper gaps: Korean Cipher falls behind English by up to 68.7 pp, whereas Cryptarithmetic over the same jamo shows no systematic penalty, and Jamo Composition accuracy predicts Korean Cipher accuracy. These contrasts are diagnostic rather than causal, consistent with difficulty in multi-step sub-syllabic execution. Korean-only tasks separate rule-application deficits, which vary in sign, from a Kinship deficit positive in all 12. Finally, a salient size measure fails to grow from Easy to Hard in 7 of 15 types, making structural size an unreliable proxy for empirical difficulty.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 38 references that are in the map. First posted on arXiv 5 Aug 2026.

Builds on

27 of 38 references.

References · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  5. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  6. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  7. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 308
  8. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 188
  9. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 186
  10. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 155
  11. 2023Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarkscited by 139
  12. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
15 more
  1. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  2. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
  3. 2024CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Koreancited by 65
  4. 2021Comparing Test Sets with Item Response Theorycited by 55
  5. 2020An Empirical Study of Tokenization Strategies for Various Korean NLP Taskscited by 33
  6. 2025SATBench: Benchmarking LLMs' Logical Reasoning via Automated Puzzle Generation from SAT Formulascited by 30
  7. 2025AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Modelscited by 29
  8. 2024CUTE: Measuring LLMs' Understanding of Their Tokenscited by 28
  9. 2025EXAONE 4.0: Unified Large Language Models Integrating Non-reasoning and Reasoning Modescited by 26
  10. 2026Multilingual Reasoning Gym: Multilingual Scaling of Procedural Reasoning Environmentscited by 5
  11. 2025Multi-Step Reasoning in Korean and the Emergent Miragecited by 4
  12. 2025EXECUTE: A Multilingual Benchmark for LLM Token Understandingcited by 4
  13. 2024Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learningcited by 3
  14. 2026Mi:dm 2.0 Korea-centric Bilingual Language Modelscited by 3
  15. 2025Redefining Evaluation Standards: A Unified Framework for Evaluating the Korean Capabilities of Language Modelscited by 1

Built on by

Not cited yet.