# Pragmatic Neural Language Modelling in Machine Translation

arXiv: https://arxiv.org/abs/1412.7119 · HTML: https://mlsanity.com/paper/1412.7119/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2014.

## Built on by (most cited first, 2 of 2)

- [Bilateral Personalized Dialogue Generation with Contrastive Learning](https://mlsanity.com/paper/2106.07857/) (2021)
- [MoNET: Tackle State Momentum via Noise-Enhanced Training for Dialogue State Tracking](https://mlsanity.com/paper/2211.05503/) (2022)

## Abstract (arXiv)

This paper presents an in-depth investigation on integrating neural language models in translation systems. Scaling neural language models is a difficult task, but crucial for real-world applications. This paper evaluates the impact on end-to-end MT quality of both new and existing scaling techniques. We show when explicitly normalising neural models is necessary and what optimisation tricks one should use in such scenarios. We also focus on scalable training algorithms and investigate noise contrastive estimation and diagonal contexts as sources for further speed improvements. We explore the trade-offs between neural models and back-off n-gram models and find that neural models make strong candidates for natural language applications in memory constrained environments, yet still lag behind traditional models in raw translation quality. We conclude with a set of recommendations one should follow to build a scalable neural language model for MT.