KORMo: Korean Open Reasoning Model for Everyone
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
This work presents the first large-scale investigation into constructing a fully open bilingual large language model (LLM) for a non-English language, specifically Korean, trained predominantly on synthetic data. We introduce KORMo-10B, a 10.8B-parameter model trained from scratch on a Korean-English corpus in which 68.74% of the Korean portion is synthetic. Through systematic experimentation, we demonstrate that synthetic data, when carefully curated with balanced linguistic coverage and diverse instruction styles, does not cause instability or degradation during large-scale pretraining. Furthermore, the model achieves performance comparable to that of contemporary open-weight multilingual baselines across a wide range of reasoning, knowledge, and instruction-following benchmarks. Our experiments reveal two key findings: (1) synthetic data can reliably sustain long-horizon pretraining without model collapse, and (2) bilingual instruction tuning enables near-native reasoning and discourse coherence in Korean. By fully releasing all components including data, code, training recipes, and logs, this work establishes a transparent framework for developing synthetic data-driven fully open models (FOMs) in low-resource settings and sets a reproducible precedent for future multilingual LLM research.
Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 59 of its 117 references that are in the map. First posted on arXiv 10 Oct 2025.
Builds on
59 of 117 references.
Closest ancestors · 2
Same direction, newest first
- 2025Qwen3 Technical Report14 May
- 2024DeepSeek-V3 Technical Report27 Dec
Other references · 57
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2020Longformer: The Long-Document Transformercited by 3,413
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
38 more
- 2022Emergent Abilities of Large Language Modelscited by 2,512
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
- 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
- 2020GLU Variants Improve Transformercited by 1,804
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
- 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
- 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
- 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
- 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
- 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
- 2023Textbooks Are All You Needcited by 569
- 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
- 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
- 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 487
- 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
- 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
- 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
- 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
- 20242 OLMo 2 Furiouscited by 340
- 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
- 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
- 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
- 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 160
- 2024Nemotron-4 340B Technical Reportcited by 144
- 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
- 2024CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Koreancited by 66
- 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 59
- 2025Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performancecited by 57
- 2025Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Datacited by 38
- 2019SocialIQA: Commonsense Reasoning about Social Interactionscited by 37
- 2024Tokenization Falling Short: On Subword Robustness in Large Language Modelscited by 20
- 2025Revisiting Overthinking in Long Chain-of-Thought from the Perspective of Self-Doubtcited by 18
- 2025LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillationcited by 14
50 most cited of 57 All 57 (JSON)
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Solar Open Technical Reportcited by 2
- 2026TREX: Tokenizer Regression for Optimal Data Mixturecited by 0
- 2026Beyond Static Benchmarks: Synthesizing Harmful Content via Persona-based Simulation for Robust Evaluationcited by 0
- 2026Opt.Gear Technical Reportcited by 0
- 2026Opt.Gear Technical Report2 Aug 2026
- 2026Beyond Static Benchmarks: Synthesizing Harmful Content via Persona-based Simulation for Robust Evaluation18 Apr 2026
- 2026TREX: Tokenizer Regression for Optimal Data Mixture20 Jan 2026
- 2026Solar Open Technical Report11 Jan 2026
Similar, not cited
Close in wording, no citation either way
- 2025KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs
- 2023HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models
- 2025Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought
- 2025Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean
- 2025Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap
- 2024HyperCLOVA X Technical Report
- 2025Multi-Step Reasoning in Korean and the Emergent Mirage
- 2026It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs