# A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation

arXiv: https://arxiv.org/abs/1905.10453 · HTML: https://mlsanity.com/paper/1905.10453/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 May 2019.

## Built on by (most cited first, 30 of 30)

- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [LLMs are Good Sign Language Translators](https://mlsanity.com/paper/2404.00925/) (2024)
- [Impact of Tokenization on Language Models: An Analysis for Turkish](https://mlsanity.com/paper/2204.08832/) (2022)
- [Robust Open-Vocabulary Translation from Visual Text Representations](https://mlsanity.com/paper/2104.08211/) (2021)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [A Formal Perspective on Byte-Pair Encoding](https://mlsanity.com/paper/2306.16837/) (2023)
- [Considerations for meaningful sign language machine translation based on glosses](https://mlsanity.com/paper/2211.15464/) (2022)
- [The Foundations of Tokenization: Statistical and Computational Concerns](https://mlsanity.com/paper/2407.11606/) (2024)
- [How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?](https://mlsanity.com/paper/2204.14268/) (2022)
- [Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal](https://mlsanity.com/paper/2404.17808/) (2024)
- [Confounding Factors in Relating Model Performance to Morphology](https://mlsanity.com/paper/2511.01380/) (2025)
- [Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speech](https://mlsanity.com/paper/2203.17190/) (2022)
- [PhoMT: A High-Quality and Large-Scale Benchmark Dataset for Vietnamese-English Machine Translation](https://mlsanity.com/paper/2110.12199/) (2021)
- [Extended Parallel Corpus for Amharic-English Machine Translation](https://mlsanity.com/paper/2104.03543/) (2021)
- [Efficient Inference for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2109.06679/) (2021)
- [CipherDAug: Ciphertext based Data Augmentation for Neural Machine Translation](https://mlsanity.com/paper/2204.00665/) (2022)
- [CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code](https://mlsanity.com/paper/2308.00683/) (2023)
- [Transformers for Low-Resource Languages: Is F'eidir Linn!](https://mlsanity.com/paper/2403.01985/) (2024)
- [Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models](https://mlsanity.com/paper/2403.11130/) (2024)
- [Multi-Sentence Resampling: A Simple Approach to Alleviate Dataset Length Bias and Beam-Search Degradation](https://mlsanity.com/paper/2109.06253/) (2021)
- [Human Evaluation of English--Irish Transformer-Based NMT](https://mlsanity.com/paper/2403.02366/) (2024)
- [Reinforcement Learning with Large Action Spaces for Neural Machine Translation](https://mlsanity.com/paper/2210.03053/) (2022)
- [Enhancing the Transformer Decoder with Transition-based Syntax](https://mlsanity.com/paper/2101.12640/) (2021)
- [MobileNMT: Enabling Translation in 15MB and 30ms](https://mlsanity.com/paper/2306.04235/) (2023)
- [The University of Edinburgh's Submission to the WMT22 Code-Mixing Shared Task (MixMT)](https://mlsanity.com/paper/2210.11309/) (2022)
- [Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealing](https://mlsanity.com/paper/2401.16055/) (2024)
- [Self-Vocabularizing Training for Neural Machine Translation](https://mlsanity.com/paper/2503.13837/) (2025)
- [Segmentation Beyond Defaults: Asymmetrical Byte Pair Encoding for Optimal Machine Translation Performance](https://mlsanity.com/paper/2511.03383/) (2025)
- [Character Iconicity vs. Arbitrariness: An Arabic NLP Perspective](https://mlsanity.com/paper/2608.02935/) (2026)

## Abstract (arXiv)

Most neural machine translation systems are built upon subword units extracted by methods such as Byte-Pair Encoding (BPE) or wordpiece. However, the choice of number of merge operations is generally made by following existing recipes. In this paper, we conduct a systematic exploration on different numbers of BPE merge operations to understand how it interacts with the model architecture, the strategy to build vocabularies and the language pair. Our exploration could provide guidance for selecting proper BPE configurations in the future. Most prominently: we show that for LSTM-based architectures, it is necessary to experiment with a wide range of different BPE operations as there is no typical optimal BPE configuration, whereas for Transformer architectures, smaller BPE size tends to be a typically optimal choice. We urge the community to make prudent choices with subword merge operations, as our experiments indicate that a sub-optimal BPE configuration alone could easily reduce the system performance by 3-4 BLEU points.