# Improving Lexical Choice in Neural Machine Translation

arXiv: https://arxiv.org/abs/1710.01329 · HTML: https://mlsanity.com/paper/1710.01329/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Oct 2017.

## Built on by (most cited first, 17 of 17)

- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation](https://mlsanity.com/paper/2106.00903/) (2021)
- [Variational Graph Normalized Auto-Encoders](https://mlsanity.com/paper/2108.08046/) (2021)
- [Lexicon Learning for Few-Shot Neural Sequence Modeling](https://mlsanity.com/paper/2106.03993/) (2021)
- [Streaming Deep Reinforcement Learning Finally Works](https://mlsanity.com/paper/2410.14606/) (2024)
- [Low-Rank Softmax Can Have Unargmaxable Classes in Theory but Rarely in Practice](https://mlsanity.com/paper/2203.06462/) (2022)
- [How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation?](https://mlsanity.com/paper/2105.12900/) (2021)
- [MoRSE: Bridging the Gap in Cybersecurity Expertise with Retrieval Augmented Generation](https://mlsanity.com/paper/2407.15748/) (2024)
- [Transformer Normalisation Layers and the Independence of Semantic Subspaces](https://mlsanity.com/paper/2406.17837/) (2024)
- [Switching to Discriminative Image Captioning by Relieving a Bottleneck of Reinforcement Learning](https://mlsanity.com/paper/2212.03230/) (2022)
- [HintedBT: Augmenting Back-Translation with Quality and Transliteration Hints](https://mlsanity.com/paper/2109.04443/) (2021)
- [Rule-based Morphological Inflection Improves Neural Terminology Translation](https://mlsanity.com/paper/2109.04620/) (2021)
- [Variance-Aware Machine Translation Test Sets](https://mlsanity.com/paper/2111.04079/) (2021)
- [Advances in Electron Microscopy with Deep Learning](https://mlsanity.com/paper/2101.01178/) (2021)
- [Optimizing Deep Transformers for Chinese-Thai Low-Resource Translation](https://mlsanity.com/paper/2212.12662/) (2022)
- [LABO: Towards Learning Optimal Label Regularization via Bi-level Optimization](https://mlsanity.com/paper/2305.04971/) (2023)
- [Massively Multilingual Text Translation For Low-Resource Languages](https://mlsanity.com/paper/2401.16582/) (2024)

## Abstract (arXiv)

We explore two solutions to the problem of mistranslating rare words in neural machine translation. First, we argue that the standard output layer, which computes the inner product of a vector representing the context with all possible output word embeddings, rewards frequent words disproportionately, and we propose to fix the norms of both vectors to a constant value. Second, we integrate a simple lexical module which is jointly trained with the rest of the model. We evaluate our approaches on eight language pairs with data sizes ranging from 100k to 8M words, and achieve improvements of up to +4.3 BLEU, surpassing phrase-based translation in nearly all settings.