# How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?

arXiv: https://arxiv.org/abs/2208.05225 · HTML: https://mlsanity.com/paper/2208.05225/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 10 Aug 2022.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [Facebook FAIR's WMT19 News Translation Task Submission](https://mlsanity.com/paper/1907.06616/) (2019)
- [On Using Very Large Target Vocabulary for Neural Machine Translation](https://mlsanity.com/paper/1412.2007/) (2014)
- [Jointly Learning to Align and Translate with Transformer Models](https://mlsanity.com/paper/1909.02074/) (2019)
- [Addressing the Rare Word Problem in Neural Machine Translation](https://mlsanity.com/paper/1410.8206/) (2014)
- [Pointing the Unknown Words](https://mlsanity.com/paper/1603.08148/) (2016)
- [Dynamic Programming Encoding for Subword Segmentation in Neural Machine Translation](https://mlsanity.com/paper/2005.06606/) (2020)
- [Character-based Neural Machine Translation](https://mlsanity.com/paper/1603.00810/) (2016)
- [Accurate Word Alignment Induction from Neural Machine Translation](https://mlsanity.com/paper/2004.14837/) (2020)
- [Revisiting Character-Based Neural Machine Translation with Capacity and Compression](https://mlsanity.com/paper/1808.09943/) (2018)
- [How Much Does Tokenization Affect Neural Machine Translation?](https://mlsanity.com/paper/1812.08621/) (2018)
- [Optimizing Transformer for Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2011.02266/) (2020)
- [Morphological Word Segmentation on Agglutinative Languages for Neural Machine Translation](https://mlsanity.com/paper/2001.01589/) (2020)
- [Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring](https://mlsanity.com/paper/1908.05925/) (2019)

## Built on by (most cited first, 3 of 3)

- [Revolutionizing Cyber Threat Detection with Large Language Models: A privacy-preserving BERT-based Lightweight Model for IoT/IIoT Devices](https://mlsanity.com/paper/2306.14263/) (2023)
- [Generating particle physics Lagrangians with transformers](https://mlsanity.com/paper/2501.09729/) (2025)
- [Impact of Domain-Adapted Multilingual Neural Machine Translation in the Medical Domain](https://mlsanity.com/paper/2212.02143/) (2022)

## Abstract (arXiv)

Neural Machine Translation (NMT) is an open vocabulary problem. As a result, dealing with the words not occurring during training (a.k.a. out-of-vocabulary (OOV) words) have long been a fundamental challenge for NMT systems. The predominant method to tackle this problem is Byte Pair Encoding (BPE) which splits words, including OOV words, into sub-word segments. BPE has achieved impressive results for a wide range of translation tasks in terms of automatic evaluation metrics. While it is often assumed that by using BPE, NMT systems are capable of handling OOV words, the effectiveness of BPE in translating OOV words has not been explicitly measured. In this paper, we study to what extent BPE is successful in translating OOV words at the word-level. We analyze the translation quality of OOV words based on word type, number of segments, cross-attention weights, and the frequency of segment n-grams in the training data. Our experiments show that while careful BPE settings seem to be fairly useful in translating OOV words across datasets, a considerable percentage of OOV words are translated incorrectly. Furthermore, we highlight the slightly higher effectiveness of BPE in translating OOV words for special cases, such as named-entities and when the languages involved are linguistically close to each other.