mlsanityData to 14 Oct 2026

KORMo: Korean Open Reasoning Model for Everyone

arXiv 2510.09426 ↗First posted 10 Oct 2025cs.CLcited by 4

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

This work presents the first large-scale investigation into constructing a fully open bilingual large language model (LLM) for a non-English language, specifically Korean, trained predominantly on synthetic data. We introduce KORMo-10B, a 10.8B-parameter model trained from scratch on a Korean-English corpus in which 68.74% of the Korean portion is synthetic. Through systematic experimentation, we demonstrate that synthetic data, when carefully curated with balanced linguistic coverage and diverse instruction styles, does not cause instability or degradation during large-scale pretraining. Furthermore, the model achieves performance comparable to that of contemporary open-weight multilingual baselines across a wide range of reasoning, knowledge, and instruction-following benchmarks. Our experiments reveal two key findings: (1) synthetic data can reliably sustain long-horizon pretraining without model collapse, and (2) bilingual instruction tuning enables near-native reasoning and discourse coherence in Korean. By fully releasing all components including data, code, training recipes, and logs, this work establishes a transparent framework for developing synthetic data-driven fully open models (FOMs) in low-resource settings and sets a reproducible precedent for future multilingual LLM research.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 59 of its 117 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

59 of 117 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2024DeepSeek-V3 Technical Report27 Dec

Other references · 57

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2020Longformer: The Long-Document Transformercited by 3,413
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
38 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,512
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  7. 2020GLU Variants Improve Transformercited by 1,804
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  11. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  12. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  13. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  14. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  15. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  16. 2023Textbooks Are All You Needcited by 569
  17. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  18. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  19. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
  20. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  21. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  22. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  23. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  24. 20242 OLMo 2 Furiouscited by 340
  25. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  26. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  27. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  28. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
  29. 2024Nemotron-4 340B Technical Reportcited by 144
  30. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
  31. 2024CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Koreancited by 66
  32. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 59
  33. 2025Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performancecited by 57
  34. 2025Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Datacited by 38
  35. 2019SocialIQA: Commonsense Reasoning about Social Interactionscited by 37
  36. 2024Tokenization Falling Short: On Subword Robustness in Large Language Modelscited by 20
  37. 2025Revisiting Overthinking in Long Chain-of-Thought from the Perspective of Self-Doubtcited by 18
  38. 2025LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillationcited by 14

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs
  2. 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models
  3. 2025Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought
  4. 2025Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean
  5. 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap
  6. 2024HyperCLOVA X Technical Report
  7. 2025Multi-Step Reasoning in Korean and the Emergent Mirage
  8. 2026It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs