# Addressing the Rare Word Problem in Neural Machine Translation

arXiv: https://arxiv.org/abs/1410.8206 · HTML: https://mlsanity.com/paper/1410.8206/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2014.

## Built on by (most cited first, 30 of 50)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [The Foundations of Tokenization: Statistical and Computational Concerns](https://mlsanity.com/paper/2407.11606/) (2024)
- [AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain](https://mlsanity.com/paper/2110.13434/) (2021)
- [Handwritten Mathematical Expression Recognition via Attention Aggregation based Bi-directional Mutual Learning](https://mlsanity.com/paper/2112.03603/) (2021)
- [Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Models](https://mlsanity.com/paper/2401.08350/) (2024)
- [Abstractive Text Summarization: State of the Art, Challenges, and Improvements](https://mlsanity.com/paper/2409.02413/) (2024)
- [Training Multilingual Pre-trained Language Model with Byte-level Subwords](https://mlsanity.com/paper/2101.09469/) (2021)
- [Reducing Position Bias in Simultaneous Machine Translation with Length-Aware Framework](https://mlsanity.com/paper/2203.09053/) (2022)
- [What is the Best Way for ChatGPT to Translate Poetry?](https://mlsanity.com/paper/2406.03450/) (2024)
- [Towards Achieving Human Parity on End-to-end Simultaneous Speech Translation via LLM Agent](https://mlsanity.com/paper/2407.21646/) (2024)
- [Fall of Giants: How popular text-based MLaaS fall against a simple evasion attack](https://mlsanity.com/paper/2104.05996/) (2021)
- [Frequency-Aware Contrastive Learning for Neural Machine Translation](https://mlsanity.com/paper/2112.14484/) (2021)
- [Neural Machine Translation for the Indigenous Languages of the Americas: An Introduction](https://mlsanity.com/paper/2306.06804/) (2023)
- [Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training](https://mlsanity.com/paper/2508.15390/) (2025)
- [End-to-End Lexically Constrained Machine Translation for Morphologically Rich Languages](https://mlsanity.com/paper/2106.12398/) (2021)
- [AutoTemplate: A Simple Recipe for Lexically Constrained Text Generation](https://mlsanity.com/paper/2211.08387/) (2022)
- [LiteLSTM Architecture for Deep Recurrent Neural Networks](https://mlsanity.com/paper/2201.11624/) (2022)
- [Assessing the Importance of Frequency versus Compositionality for Subword-based Tokenization in NMT](https://mlsanity.com/paper/2306.01393/) (2023)
- [Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation](https://mlsanity.com/paper/2403.16771/) (2024)
- [Family of Origin and Family of Choice: Massively Parallel Lexiconized Iterative Pretraining for Severely Low Resource Machine Translation](https://mlsanity.com/paper/2104.05848/) (2021)
- [Path sampling of recurrent neural networks by incorporating known physics](https://mlsanity.com/paper/2203.00597/) (2022)
- [Impact of Decoding Methods on Human Alignment of Conversational LLMs](https://mlsanity.com/paper/2407.19526/) (2024)
- [Neighbors Are Not Strangers: Improving Non-Autoregressive Translation under Low-Frequency Lexical Constraints](https://mlsanity.com/paper/2204.13355/) (2022)
- [How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?](https://mlsanity.com/paper/2208.05225/) (2022)
- [Neural Machine Translation for Code Generation](https://mlsanity.com/paper/2305.13504/) (2023)
- [Neural Machine Translation with Dynamic Graph Convolutional Decoder](https://mlsanity.com/paper/2305.17698/) (2023)
- [Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue](https://mlsanity.com/paper/2306.03652/) (2023)
- [Audience-specific Explanations for Machine Translation](https://mlsanity.com/paper/2309.12998/) (2023)
- [Survey of Natural Language Processing for Education: Taxonomy, Systematic Review, and Future Trends](https://mlsanity.com/paper/2401.07518/) (2024)

## Abstract (arXiv)

Neural Machine Translation (NMT) is a new approach to machine translation that has shown promising results that are comparable to traditional approaches. A significant weakness in conventional NMT systems is their inability to correctly translate very rare words: end-to-end NMTs tend to have relatively small vocabularies with a single unk symbol that represents every possible out-of-vocabulary (OOV) word. In this paper, we propose and implement an effective technique to address this problem. We train an NMT system on data that is augmented by the output of a word alignment algorithm, allowing the NMT system to emit, for each OOV word in the target sentence, the position of its corresponding word in the source sentence. This information is later utilized in a post-processing step that translates every OOV word using a dictionary. Our experiments on the WMT14 English to French translation task show that this method provides a substantial improvement of up to 2.8 BLEU points over an equivalent NMT system that does not use this technique. With 37.5 BLEU points, our NMT system is the first to surpass the best result achieved on a WMT14 contest task.