# Efficient Inference For Neural Machine Translation

arXiv: https://arxiv.org/abs/2010.02416 · HTML: https://mlsanity.com/paper/2010.02416/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 7 of 7)

- [When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute](https://mlsanity.com/paper/2102.12459/) (2021)
- [ChartSumm: A Comprehensive Benchmark for Automatic Chart Summarization of Long and Short Summaries](https://mlsanity.com/paper/2304.13620/) (2023)
- [Efficient Inference for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2109.06679/) (2021)
- [Bridging Cross-Lingual Gaps During Leveraging the Multilingual Sequence-to-Sequence Pretraining for Text Generation and Understanding](https://mlsanity.com/paper/2204.07834/) (2022)
- [Building Multilingual Machine Translation Systems That Serve Arbitrary X-Y Translations](https://mlsanity.com/paper/2206.14982/) (2022)
- [Bag of Tricks for Optimizing Transformer Efficiency](https://mlsanity.com/paper/2109.04030/) (2021)
- [Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translation](https://mlsanity.com/paper/2407.01126/) (2024)

## Abstract (arXiv)

Large Transformer models have achieved state-of-the-art results in neural machine translation and have become standard in the field. In this work, we look for the optimal combination of known techniques to optimize inference speed without sacrificing translation quality. We conduct an empirical study that stacks various approaches and demonstrates that combination of replacing decoder self-attention with simplified recurrent units, adopting a deep encoder and a shallow decoder architecture and multi-head attention pruning can achieve up to 109% and 84% speedup on CPU and GPU respectively and reduce the number of parameters by 25% while maintaining the same translation quality in terms of BLEU.