# On Using Very Large Target Vocabulary for Neural Machine Translation

arXiv: https://arxiv.org/abs/1412.2007 · HTML: https://mlsanity.com/paper/1412.2007/

## Key facts

- Cited by 80 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2014.

## Built on by (most cited first, 30 of 80)

- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [A Cookbook of Self-Supervised Learning](https://mlsanity.com/paper/2304.12210/) (2023)
- [Hash Layers For Large Sparse Models](https://mlsanity.com/paper/2106.04426/) (2021)
- [Algorithms to estimate Shapley value feature attributions](https://mlsanity.com/paper/2207.07605/) (2022)
- [Tokenization and the Noiseless Channel](https://mlsanity.com/paper/2306.16842/) (2023)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Sparse-Interest Network for Sequential Recommendation](https://mlsanity.com/paper/2102.09267/) (2021)
- [Improving Deep Learning for HAR with shallow LSTMs](https://mlsanity.com/paper/2108.00702/) (2021)
- [gSASRec: Reducing Overconfidence in Sequential Recommendation Trained with Negative Sampling](https://mlsanity.com/paper/2308.07192/) (2023)
- [SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining](https://mlsanity.com/paper/2108.08983/) (2021)
- [Learning to Generate Image Embeddings with User-level Differential Privacy](https://mlsanity.com/paper/2211.10844/) (2022)
- [Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond](https://mlsanity.com/paper/2310.05317/) (2023)
- [An Overview and Experimental Study of Learning-based Optimization Algorithms for Vehicle Routing Problem](https://mlsanity.com/paper/2107.07076/) (2021)
- [Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training](https://mlsanity.com/paper/2109.07306/) (2021)
- [The Foundations of Tokenization: Statistical and Computational Concerns](https://mlsanity.com/paper/2407.11606/) (2024)
- [DimeRec: A Unified Framework for Enhanced Sequential Recommendation via Generative Diffusion Models](https://mlsanity.com/paper/2408.12153/) (2024)
- [Emotion-aware Chat Machine: Automatic Emotional Response Generation for Human-like Emotional Interaction](https://mlsanity.com/paper/2106.03044/) (2021)
- [BFClass: A Backdoor-free Text Classification Framework](https://mlsanity.com/paper/2109.10855/) (2021)
- [Training Multilingual Pre-trained Language Model with Byte-level Subwords](https://mlsanity.com/paper/2101.09469/) (2021)
- [Dynamic Memory based Attention Network for Sequential Recommendation](https://mlsanity.com/paper/2102.09269/) (2021)
- [Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs](https://mlsanity.com/paper/2503.16870/) (2025)
- [DKPLM: Decomposable Knowledge-enhanced Pre-trained Language Model for Natural Language Understanding](https://mlsanity.com/paper/2112.01047/) (2021)
- [Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecycle](https://mlsanity.com/paper/2303.01980/) (2023)
- [Effects of sub-word segmentation on performance of transformer language models](https://mlsanity.com/paper/2305.05480/) (2023)
- [Knowledge Prompting in Pre-trained Language Model for Natural Language Understanding](https://mlsanity.com/paper/2210.08536/) (2022)
- [MAKE: Vision-Language Pre-training based Product Retrieval in Taobao Search](https://mlsanity.com/paper/2301.12646/) (2023)
- [Lookaround Optimizer: k steps around, 1 step average](https://mlsanity.com/paper/2306.07684/) (2023)
- [Expressivity and Approximation Properties of Deep Neural Networks with ReLUk Activation](https://mlsanity.com/paper/2312.16483/) (2023)
- [Fast Variational AutoEncoder with Inverted Multi-Index for Collaborative Filtering](https://mlsanity.com/paper/2109.05773/) (2021)
- [GiGL: Large-Scale Graph Neural Networks at Snapchat](https://mlsanity.com/paper/2502.15054/) (2025)

## Abstract (arXiv)

Neural machine translation, a recently proposed approach to machine translation based purely on neural networks, has shown promising results compared to the existing approaches such as phrase-based statistical machine translation. Despite its recent success, neural machine translation has its limitation in handling a larger vocabulary, as training complexity as well as decoding complexity increase proportionally to the number of target words. In this paper, we propose a method that allows us to use a very large target vocabulary without increasing training complexity, based on importance sampling. We show that decoding can be efficiently done even with the model having a very large target vocabulary by selecting only a small subset of the whole target vocabulary. The models trained by the proposed approach are empirically found to outperform the baseline models with a small vocabulary as well as the LSTM-based neural machine translation models. Furthermore, when we use the ensemble of a few models with very large target vocabularies, we achieve the state-of-the-art translation performance (measured by BLEU) on the English->German translation and almost as high performance as state-of-the-art English->French translation system.