# Modeling Coverage for Non-Autoregressive Neural Machine Translation

arXiv: https://arxiv.org/abs/2104.11897 · HTML: https://mlsanity.com/paper/2104.11897/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving.
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 24 Apr 2021.

## Placed in

- Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/)

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [Modeling Coverage for Neural Machine Translation](https://mlsanity.com/paper/1601.04811/) (2016)
- [Fast Decoding in Sequence Models using Discrete Latent Variables](https://mlsanity.com/paper/1803.03382/) (2018)
- [FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow](https://mlsanity.com/paper/1909.02480/) (2019)
- [End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification](https://mlsanity.com/paper/1811.04719/) (2018)
- [Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior](https://mlsanity.com/paper/1908.07181/) (2019)
- [Non-Autoregressive Machine Translation with Auxiliary Regularization](https://mlsanity.com/paper/1902.10245/) (2019)
- [Semi-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1808.08583/) (2018)
- [Regularizing Neural Machine Translation by Target-bidirectional Agreement](https://mlsanity.com/paper/1808.04064/) (2018)
- [Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1911.09320/) (2019)
- [Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input](https://mlsanity.com/paper/1812.09664/) (2018)
- [Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information](https://mlsanity.com/paper/1911.02215/) (2019)
- [Neural Machine Translation with Reconstruction](https://mlsanity.com/paper/1611.01874/) (2016)
- [Hint-Based Training for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/1909.06708/) (2019)
- [Consistency by Agreement in Zero-shot Neural Machine Translation](https://mlsanity.com/paper/1904.02338/) (2019)
- [Syntactically Supervised Transformers for Faster Neural Machine Translation](https://mlsanity.com/paper/1906.02780/) (2019)
- [Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2006.05165/) (2020)
- [Non-autoregressive Transformer by Position Learning](https://mlsanity.com/paper/1911.10677/) (2019)
- [Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1906.09444/) (2019)
- [Modeling Past and Future for Neural Machine Translation](https://mlsanity.com/paper/1711.09502/) (2017)

## Built on by (most cited first, 4 of 4)

- [One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation](https://mlsanity.com/paper/2205.14333/) (2022)
- [Sequence-Level Training for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2106.08122/) (2021)
- [Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.03953/) (2022)
- [Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.05193/) (2022)

## Abstract (arXiv)

Non-Autoregressive Neural Machine Translation (NAT) has achieved significant inference speedup by generating all tokens simultaneously. Despite its high efficiency, NAT usually suffers from two kinds of translation errors: over-translation (e.g. repeated tokens) and under-translation (e.g. missing translations), which eventually limits the translation quality. In this paper, we argue that these issues of NAT can be addressed through coverage modeling, which has been proved to be useful in autoregressive decoding. We propose a novel Coverage-NAT to model the coverage information directly by a token-level coverage iterative refinement mechanism and a sentence-level coverage agreement, which can remind the model if a source token has been translated or not and improve the semantics consistency between the translation and the source, respectively. Experimental results on WMT14 En-De and WMT16 En-Ro translation tasks show that our method can alleviate those errors and achieve strong improvements over the baseline system.