# RedWhale: An Adapted Korean LLM Through Efficient Continual Pretraining

arXiv: https://arxiv.org/abs/2408.11294 · HTML: https://mlsanity.com/paper/2408.11294/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2024.

## Built on by (most cited first, 8 of 8)

- [Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus](https://mlsanity.com/paper/2410.14815/) (2024)
- [Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resources](https://mlsanity.com/paper/2504.04152/) (2025)
- [BgGPT 1.0: Extending English-centric LLMs to other languages](https://mlsanity.com/paper/2412.10893/) (2024)
- [Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMs](https://mlsanity.com/paper/2412.14471/) (2024)
- [Improving Multilingual Capabilities with Cultural and Local Knowledge in Large Language Models While Enhancing Native Performance](https://mlsanity.com/paper/2504.09753/) (2025)
- [Optimizing Korean-Centric LLMs via Token Pruning](https://mlsanity.com/paper/2604.16235/) (2026)
- [HanjaBridge: Resolving Semantic Ambiguity in Korean LLMs via Hanja-Augmented Pre-Training](https://mlsanity.com/paper/2507.10920/) (2025)
- [Beyond Initialization Loss: A Systematic Study of Token Embedding Initialization Strategies for LLM Vocabulary Extension](https://mlsanity.com/paper/2608.03494/) (2026)

## Abstract (arXiv)

The field of Natural Language Processing (NLP) has seen significant advancements with the development of Large Language Models (LLMs). However, much of this research remains focused on English, often overlooking low-resource languages like Korean. This oversight presents challenges due to the unique non-alphabetic token structure of Korean and the substantial memory and computational demands required for LLM training, which frequently lead to memory constraints and out-of-memory errors. To address these issues, we present RedWhale, a model specifically tailored for Korean language processing. RedWhale is developed using an efficient continual pretraining approach that includes a comprehensive Korean corpus preprocessing pipeline, a specialized tokenizer, an optimized model initialization technique, and a multistage pretraining strategy. These innovations collectively reduce training time and computational costs while maintaining high levels of accuracy and comprehension. By leveraging cross-lingual transfer learning, which exploits shared linguistic similarities across languages, RedWhale builds on English models to enhance Korean language processing. Experimental results demonstrate that RedWhale outperforms other leading models on Korean NLP benchmarks, including the Korean Balanced Evaluation of Significant Tasks (KoBEST), showing superior understanding and generation of Korean text. Furthermore, RedWhale showed no signs of convergence even after pretraining on 9.7 billion tokens, indicating the potential for further improvements with additional training. This work represents a significant advancement in bridging the linguistic divide, particularly in enhancing NLP capabilities for the Korean language.