# A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models

arXiv: https://arxiv.org/abs/2306.02254 · HTML: https://mlsanity.com/paper/2306.02254/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 4 Jun 2023.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [KOBEST: Korean Balanced Evaluation of Significant Tasks](https://mlsanity.com/paper/2204.04541/) (2022)

## Built on by (most cited first, 22 of 22)

- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [KMMLU: Measuring Massive Multitask Language Understanding in Korean](https://mlsanity.com/paper/2402.11548/) (2024)
- [CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean](https://mlsanity.com/paper/2403.06412/) (2024)
- [HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models](https://mlsanity.com/paper/2309.02706/) (2023)
- [KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge](https://mlsanity.com/paper/2402.13605/) (2024)
- [HyperCLOVA X Technical Report](https://mlsanity.com/paper/2404.01954/) (2024)
- [Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models](https://mlsanity.com/paper/2402.14714/) (2024)
- [Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean](https://mlsanity.com/paper/2403.10882/) (2024)
- [TeenyTinyLlama: open-source tiny language models trained in Brazilian Portuguese](https://mlsanity.com/paper/2401.16640/) (2024)
- [BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism](https://mlsanity.com/paper/2412.17933/) (2024)
- [Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindi](https://mlsanity.com/paper/2504.06011/) (2025)
- [Orion-14B: Open-source Multilingual Large Language Models](https://mlsanity.com/paper/2401.12246/) (2024)
- [Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models](https://mlsanity.com/paper/2410.08731/) (2024)
- [Trillion 7B Technical Report](https://mlsanity.com/paper/2504.15431/) (2025)
- [KoDialogBench: Evaluating Conversational Understanding of Language Models with Korean Dialogue Benchmark](https://mlsanity.com/paper/2402.17377/) (2024)
- [Automatic Construction of a Korean Toxic Instruction Dataset for Ethical Tuning of Large Language Models](https://mlsanity.com/paper/2311.18215/) (2023)
- [Thunder-DeID: Accurate and Efficient De-identification Framework for Korean Court Judgments](https://mlsanity.com/paper/2506.15266/) (2025)
- [DaG LLM ver 1.0: Pioneering Instruction-Tuned Language Modeling for Korean NLP](https://mlsanity.com/paper/2311.13784/) (2023)
- [K-Act2Emo: Korean Commonsense Knowledge Graph for Indirect Emotional Expression](https://mlsanity.com/paper/2403.14253/) (2024)
- [KIT-19: A Comprehensive Korean Instruction Toolkit on 19 Tasks for Fine-Tuning Korean Large Language Models](https://mlsanity.com/paper/2403.16444/) (2024)
- [GECKO: Generative Language Model for English, Code and Korean](https://mlsanity.com/paper/2405.15640/) (2024)
- [Llamion Technical Report](https://mlsanity.com/paper/2605.25676/) (2026)

## Abstract (arXiv)

Polyglot is a pioneering project aimed at enhancing the non-English language performance of multilingual language models. Despite the availability of various multilingual models such as mBERT (Devlin et al., 2019), XGLM (Lin et al., 2022), and BLOOM (Scao et al., 2022), researchers and developers often resort to building monolingual models in their respective languages due to the dissatisfaction with the current multilingual models non-English language capabilities. Addressing this gap, we seek to develop advanced multilingual language models that offer improved performance in non-English languages. In this paper, we introduce the Polyglot Korean models, which represent a specific focus rather than being multilingual in nature. In collaboration with TUNiB, our team collected 1.2TB of Korean data meticulously curated for our research journey. We made a deliberate decision to prioritize the development of Korean models before venturing into multilingual models. This choice was motivated by multiple factors: firstly, the Korean models facilitated performance comparisons with existing multilingual models; and finally, they catered to the specific needs of Korean companies and researchers. This paper presents our work in developing the Polyglot Korean models, which propose some steps towards addressing the non-English language performance gap in multilingual language models.