# How Much Does Tokenization Affect Neural Machine Translation?

arXiv: https://arxiv.org/abs/1812.08621 · HTML: https://mlsanity.com/paper/1812.08621/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Dec 2018.

## Built on by (most cited first, 26 of 26)

- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Models](https://mlsanity.com/paper/2212.10769/) (2022)
- [The SIGMORPHON 2022 Shared Task on Morpheme Segmentation](https://mlsanity.com/paper/2206.07615/) (2022)
- [The Foundations of Tokenization: Statistical and Computational Concerns](https://mlsanity.com/paper/2407.11606/) (2024)
- [How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?](https://mlsanity.com/paper/2204.14268/) (2022)
- [Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologies](https://mlsanity.com/paper/2312.11779/) (2023)
- [Joint Optimization of Tokenization and Downstream Model](https://mlsanity.com/paper/2105.12410/) (2021)
- [FLAME: A small language model for spreadsheet formulas](https://mlsanity.com/paper/2301.13779/) (2023)
- [Beyond Text Compression: Evaluating Tokenizers Across Scales](https://mlsanity.com/paper/2506.03101/) (2025)
- [MorphPiece : A Linguistic Tokenizer for Large Language Models](https://mlsanity.com/paper/2307.07262/) (2023)
- [Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark](https://mlsanity.com/paper/2502.07057/) (2025)
- [CompoundPiece: Evaluating and Improving Decompounding Performance of Language Models](https://mlsanity.com/paper/2305.14214/) (2023)
- [Exploring the Impact of Training Data Distribution and Subword Tokenization on Gender Bias in Machine Translation](https://mlsanity.com/paper/2309.12491/) (2023)
- [Improving Multilingual Neural Machine Translation System for Indic Languages](https://mlsanity.com/paper/2209.13279/) (2022)
- [Low-Resourced Machine Translation for Senegalese Wolof Language](https://mlsanity.com/paper/2305.00606/) (2023)
- [Beqi: Revitalize the Senegalese Wolof Language with a Robust Spelling Corrector](https://mlsanity.com/paper/2305.08518/) (2023)
- [Tokenization with Factorized Subword Encoding](https://mlsanity.com/paper/2306.07764/) (2023)
- [Dougal Dil .Icslemede Tokenizasyon Standartlari ve "Olcc"um"u: T"urkcce "Uzerinden B"uy"uk Dil Modellerinin Karcsilacstirmali Analizi](https://mlsanity.com/paper/2508.13058/) (2025)
- [How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?](https://mlsanity.com/paper/2208.05225/) (2022)
- [Revisiting subword tokenization: A case study on affixal negation in large language models](https://mlsanity.com/paper/2404.02421/) (2024)
- [Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew](https://mlsanity.com/paper/2602.05648/) (2026)
- [Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet](https://mlsanity.com/paper/2606.20993/) (2026)
- [Pointer-Generator Networks for Low-Resource Machine Translation: Don't Copy That!](https://mlsanity.com/paper/2403.10963/) (2024)
- [Morphological Typology in BPE Subword Productivity and Language Modeling](https://mlsanity.com/paper/2410.23656/) (2024)
- [Unsupervised Morphological Tree Tokenizer](https://mlsanity.com/paper/2406.15245/) (2024)
- [Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-service](https://mlsanity.com/paper/2506.06446/) (2025)

## Abstract (arXiv)

Tokenization or segmentation is a wide concept that covers simple processes such as separating punctuation from words, or more sophisticated processes such as applying morphological knowledge. Neural Machine Translation (NMT) requires a limited-size vocabulary for computational cost and enough examples to estimate word embeddings. Separating punctuation and splitting tokens into words or subwords has proven to be helpful to reduce vocabulary and increase the number of examples of each word, improving the translation quality. Tokenization is more challenging when dealing with languages with no separator between words. In order to assess the impact of the tokenization in the quality of the final translation on NMT, we experimented on five tokenizers over ten language pairs. We reached the conclusion that the tokenization significantly affects the final translation quality and that the best tokenizer differs for different language pairs.