# An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks

arXiv: https://arxiv.org/abs/2010.02534 · HTML: https://mlsanity.com/paper/2010.02534/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 30 of 33)

- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [KLUE: Korean Language Understanding Evaluation](https://mlsanity.com/paper/2105.09680/) (2021)
- [What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers](https://mlsanity.com/paper/2109.04650/) (2021)
- [Impact of Tokenization on Language Models: An Analysis for Turkish](https://mlsanity.com/paper/2204.08832/) (2022)
- [Language Modelling with Pixels](https://mlsanity.com/paper/2207.06991/) (2022)
- [Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark](https://mlsanity.com/paper/2405.20574/) (2024)
- [Evaluating Morphological Alignment of Tokenizers in 70 Languages](https://mlsanity.com/paper/2507.06378/) (2025)
- [How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese](https://mlsanity.com/paper/2306.09572/) (2023)
- [Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge](https://mlsanity.com/paper/2404.13292/) (2024)
- [KatFishNet: Detecting LLM-Generated Korean Text through Linguistic Feature Analysis](https://mlsanity.com/paper/2503.00032/) (2025)
- [Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models](https://mlsanity.com/paper/2403.11130/) (2024)
- [Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement](https://mlsanity.com/paper/2403.13754/) (2024)
- [A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models](https://mlsanity.com/paper/2310.12936/) (2023)
- [Language Surgery in Multilingual Large Language Models](https://mlsanity.com/paper/2506.12450/) (2025)
- [A Survey on Awesome Korean NLP Datasets](https://mlsanity.com/paper/2112.01624/) (2021)
- [Can Perplexity Predict Fine-tuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for Nepali](https://mlsanity.com/paper/2404.18071/) (2024)
- [From Words to Music: A Study of Subword Tokenization Techniques in Symbolic Music Generation](https://mlsanity.com/paper/2304.08953/) (2023)
- [Improving Korean NLP Tasks with Linguistically Informed Subword Tokenization and Sub-character Decomposition](https://mlsanity.com/paper/2311.03928/) (2023)
- [KpopMT: Translation Dataset with Terminology for Kpop Fandom](https://mlsanity.com/paper/2407.07413/) (2024)
- [Less Is More: Reducing Token Counts Without Compromising Performance](https://mlsanity.com/paper/2506.15138/) (2025)
- [FREED++: Improving RL Agents for Fragment-Based Molecule Generation by Thorough Reproduction](https://mlsanity.com/paper/2401.09840/) (2024)
- [Thunder-DeID: Accurate and Efficient De-identification Framework for Korean Court Judgments](https://mlsanity.com/paper/2506.15266/) (2025)
- [Korean-English Machine Translation with Multiple Tokenization Strategy](https://mlsanity.com/paper/2105.14274/) (2021)
- [Korean Tokenization for Beam Search Rescoring in Speech Recognition](https://mlsanity.com/paper/2203.03583/) (2022)
- [Data-Driven Approach for Formality-Sensitive Machine Translation: Language-Specific Handling and Synthetic Data Generation](https://mlsanity.com/paper/2306.14514/) (2023)
- [Exploring acceptance of autonomous vehicle policies using KeyBERT and SNA: Targeting engineering students](https://mlsanity.com/paper/2307.09014/) (2023)
- [Comparative analysis of subword tokenization approaches for Indian languages](https://mlsanity.com/paper/2505.16868/) (2025)
- [Handling Korean Out-of-Vocabulary Words with Phoneme Representation Learning](https://mlsanity.com/paper/2507.04018/) (2025)
- [KITE: A Benchmark for Evaluating Korean Instruction-Following Abilities in Large Language Models](https://mlsanity.com/paper/2510.15558/) (2025)
- [LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptation](https://mlsanity.com/paper/2603.22629/) (2026)

## Abstract (arXiv)

Typically, tokenization is the very first step in most text processing works. As a token serves as an atomic unit that embeds the contextual information of text, how to define a token plays a decisive role in the performance of a model.Even though Byte Pair Encoding (BPE) has been considered the de facto standard tokenization method due to its simplicity and universality, it still remains unclear whether BPE works best across all languages and tasks. In this paper, we test several tokenization strategies in order to answer our primary research question, that is, "What is the best tokenization strategy for Korean NLP tasks?" Experimental results demonstrate that a hybrid approach of morphological segmentation followed by BPE works best in Korean to/from English machine translation and natural language understanding tasks such as KorNLI, KorSTS, NSMC, and PAWS-X. As an exception, for KorQuAD, the Korean extension of SQuAD, BPE segmentation turns out to be the most effective.