# Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study

arXiv: https://arxiv.org/abs/2509.03972 · HTML: https://mlsanity.com/paper/2509.03972/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 18 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [LLaMA Pro: Progressive LLaMA with Block Expansion](https://mlsanity.com/paper/2401.02415/) (2024)
- [NEFTune: Noisy Embeddings Improve Instruction Finetuning](https://mlsanity.com/paper/2310.05914/) (2023)
- [KMMLU: Measuring Massive Multitask Language Understanding in Korean](https://mlsanity.com/paper/2402.11548/) (2024)
- [Staged Training for Transformer Language Models](https://mlsanity.com/paper/2203.06211/) (2022)
- [Masked Structural Growth for 2x Faster Language Model Pre-training](https://mlsanity.com/paper/2305.02869/) (2023)
- [KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations](https://mlsanity.com/paper/2403.01469/) (2024)
- [EXAONE 3.0 7.8B Instruction Tuned Language Model](https://mlsanity.com/paper/2408.03541/) (2024)
- [HyperCLOVA X Technical Report](https://mlsanity.com/paper/2404.01954/) (2024)

## Built on by (most cited first, 2 of 2)

- [Fundamental Reasoning Paradigms Induce Out-of-Domain Generalization in Language Models](https://mlsanity.com/paper/2602.08658/) (2026)
- [Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages](https://mlsanity.com/paper/2512.08777/) (2025)

## Abstract (arXiv)

We introduce Llama-3-Motif, a language model consisting of 102 billion parameters, specifically designed to enhance Korean capabilities while retaining strong performance in English. Developed on the Llama 3 architecture, Llama-3-Motif employs advanced training techniques, including LlamaPro and Masked Structure Growth, to effectively scale the model without altering its core Transformer architecture. Using the MoAI platform for efficient training across hyperscale GPU clusters, we optimized Llama-3-Motif using a carefully curated dataset that maintains a balanced ratio of Korean and English data. Llama-3-Motif shows decent performance on Korean-specific benchmarks, outperforming existing models and achieving results comparable to GPT-4.