# The Devil is in the Details: On the Pitfalls of Vocabulary Selection in Neural Machine Translation

arXiv: https://arxiv.org/abs/2205.06618 · HTML: https://mlsanity.com/paper/2205.06618/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 13 May 2022.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation](https://mlsanity.com/paper/2006.10369/) (2020)
- [On Using Very Large Target Vocabulary for Neural Machine Translation](https://mlsanity.com/paper/1412.2007/) (2014)
- [Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms](https://mlsanity.com/paper/1805.09843/) (2018)
- [Examining the Tip of the Iceberg: A Data Set for Idiom Translation](https://mlsanity.com/paper/1802.04681/) (2018)
- [Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translation](https://mlsanity.com/paper/2106.00169/) (2021)
- [Accelerating Neural Transformer via an Average Attention Network](https://mlsanity.com/paper/1805.00631/) (2018)
- [Marian: Cost-effective High-Quality Neural Machine Translation in C++](https://mlsanity.com/paper/1805.12096/) (2018)
- [Mask-Align: Self-Supervised Neural Word Alignment](https://mlsanity.com/paper/2012.07162/) (2020)
- [The Sockeye 2 Neural Machine Translation Toolkit at AMTA 2020](https://mlsanity.com/paper/2008.04885/) (2020)
- [Learning to Screen for Fast Softmax Inference on Large Vocabulary Neural Networks](https://mlsanity.com/paper/1810.12406/) (2018)
- [Efficient Inference for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2109.06679/) (2021)
- [Vocabulary Manipulation for Neural Machine Translation](https://mlsanity.com/paper/1605.03209/) (2016)
- [Vocabulary Selection Strategies for Neural Machine Translation](https://mlsanity.com/paper/1610.00072/) (2016)
- [The Highs and Lows of Simple Lexical Domain Adaptation Approaches for Neural Machine Translation](https://mlsanity.com/paper/2101.00421/) (2021)
- [Neural Machine Translation with Word Predictions](https://mlsanity.com/paper/1708.01771/) (2017)
- [Attention-based Vocabulary Selection for NMT Decoding](https://mlsanity.com/paper/1706.03824/) (2017)

## Built on by (most cited first, 3 of 3)

- [Sockeye 3: Fast Neural Machine Translation with PyTorch](https://mlsanity.com/paper/2207.05851/) (2022)
- [The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics](https://mlsanity.com/paper/2311.09709/) (2023)
- [PolyFrame at MWE-2026 AdMIRe 2: When Words Are Not Enough: Multimodal Idiom Disambiguation](https://mlsanity.com/paper/2602.18652/) (2026)

## Abstract (arXiv)

Vocabulary selection, or lexical shortlisting, is a well-known technique to improve latency of Neural Machine Translation models by constraining the set of allowed output words during inference. The chosen set is typically determined by separately trained alignment model parameters, independent of the source-sentence context at inference time. While vocabulary selection appears competitive with respect to automatic quality metrics in prior work, we show that it can fail to select the right set of output words, particularly for semantically non-compositional linguistic phenomena such as idiomatic expressions, leading to reduced translation quality as perceived by humans. Trading off latency for quality by increasing the size of the allowed set is often not an option in real-world scenarios. We propose a model of vocabulary selection, integrated into the neural translation model, that predicts the set of allowed output words from contextualized encoder representations. This restores translation quality of an unconstrained system, as measured by human evaluations on WMT newstest2020 and idiomatic expressions, at an inference latency competitive with alignment-based selection using aggressive thresholds, thereby removing the dependency on separately trained alignment models.