# Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models

arXiv: https://arxiv.org/abs/2402.14714 · HTML: https://mlsanity.com/paper/2402.14714/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 22 Feb 2024.

## Builds on (most cited first, 17 of 17)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [The Falcon Series of Open Language Models](https://mlsanity.com/paper/2311.16867/) (2023)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling](https://mlsanity.com/paper/2312.15166/) (2023)
- [LLaMA Beyond English: An Empirical Study on Language Capability Transfer](https://mlsanity.com/paper/2401.01055/) (2024)
- [Multilingual Large Language Models Are Not (Yet) Code-Switchers](https://mlsanity.com/paper/2305.14235/) (2023)
- [Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective](https://mlsanity.com/paper/2310.10226/) (2023)
- [KOBEST: Korean Balanced Evaluation of Significant Tasks](https://mlsanity.com/paper/2204.04541/) (2022)
- [A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Models](https://mlsanity.com/paper/2306.02254/) (2023)
- [Improving Low Compute Language Modeling with In-Domain Embedding Initialisation](https://mlsanity.com/paper/2009.14109/) (2020)

## Built on by (most cited first, 15 of 15)

- [From Tokens to Words: On the Inner Lexicon of LLMs](https://mlsanity.com/paper/2410.05864/) (2024)
- [LLM-as-a-Judge & Reward Model: What They Can and Cannot Do](https://mlsanity.com/paper/2409.11239/) (2024)
- [Sailor: Open Language Models for South-East Asia](https://mlsanity.com/paper/2404.03608/) (2024)
- [Multilingual Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2411.11072/) (2024)
- [Continually Adding New Languages to Multilingual Language Models](https://mlsanity.com/paper/2509.11414/) (2025)
- [Adapting Chat Language Models Using Only Target Unlabeled Language Data](https://mlsanity.com/paper/2412.11704/) (2024)
- [zip2zip: Inference-Time Adaptive Tokenization via Online Compression](https://mlsanity.com/paper/2506.01084/) (2025)
- [Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansion](https://mlsanity.com/paper/2604.16656/) (2026)
- [In-Place Tokenizer Expansion for Pre-trained LLMs](https://mlsanity.com/paper/2607.15232/) (2026)
- [Writing-System-Level Tokenizer Adaptation for Byte-Level BPE](https://mlsanity.com/paper/2608.00582/) (2026)
- [HyperCLOVA X 32B Think](https://mlsanity.com/paper/2601.03286/) (2026)
- [Large Byte Model: Teaching Language Models About Compiled Code](https://mlsanity.com/paper/2606.02834/) (2026)
- [CantoneseLLM v2: Reasoning in a Low-Resource Language](https://mlsanity.com/paper/2609.06970/) (2026)
- [zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Loss](https://mlsanity.com/paper/2610.13605/) (2026)
- [ixi-sLM: On-Device sLM for AI Call Agent Service](https://mlsanity.com/paper/2610.19382/) (2026)

## Abstract (arXiv)

This report introduces textttEEVE-Korean-v1.0, a Korean adaptation of large language models that exhibit remarkable capabilities across English and Korean text understanding. Building on recent highly capable but English-centric LLMs, such as SOLAR-10.7B and Phi-2, where non-English texts are inefficiently processed with English-centric tokenizers, we present an efficient and effective vocabulary expansion (EEVE) method, which encompasses parameter freezing and subword initialization. In contrast to previous efforts that believe new embeddings require trillions of training tokens, we show that our method can significantly boost non-English proficiency within just 2 billion tokens. Surpassing most instruction-tuned LLMs on the Open Ko-LLM Leaderboard, as of January 2024, our model textttEEVE-Korean-10.8B-v1.0 ranks as the leading Korean pre-trained model in the open-source community, according to Hugging Face's leaderboard. We open-source our models on Huggingface to empower the open research community in various languages.