mlsanityData to 7 Oct 2026

Modeling Coverage for Non-Autoregressive Neural Machine Translation

arXiv 2104.11897 ↗First posted 24 Apr 2021cs.CLcited by 4

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Non-Autoregressive Neural Machine Translation (NAT) has achieved significant inference speedup by generating all tokens simultaneously. Despite its high efficiency, NAT usually suffers from two kinds of translation errors: over-translation (e.g. repeated tokens) and under-translation (e.g. missing translations), which eventually limits the translation quality. In this paper, we argue that these issues of NAT can be addressed through coverage modeling, which has been proved to be useful in autoregressive decoding. We propose a novel Coverage-NAT to model the coverage information directly by a token-level coverage iterative refinement mechanism and a sentence-level coverage agreement, which can remind the model if a source token has been translated or not and improve the semantics consistency between the translation and the source, respectively. Experimental results on WMT14 En-De and WMT16 En-Ro translation tasks show that our method can alleviate those errors and achieve strong improvements over the baseline system.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 32 of its 40 references that are in the map. First posted on arXiv 24 Apr 2021.

Builds on

32 of 40 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  8. 2016Sequence-Level Knowledge Distillationcited by 761
  9. 2017Non-Autoregressive Neural Machine Translationcited by 457
  10. 2019Levenshtein Transformercited by 197
  11. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  12. 2016Modeling Coverage for Neural Machine Translationcited by 110
20 more
  1. 2018Fast Decoding in Sequence Models using Discrete Latent Variablescited by 103
  2. 2019FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flowcited by 89
  3. 2018End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classificationcited by 72
  4. 2019Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posteriorcited by 58
  5. 2019Non-Autoregressive Machine Translation with Auxiliary Regularizationcited by 52
  6. 2018Semi-Autoregressive Neural Machine Translationcited by 48
  7. 2018Regularizing Neural Machine Translation by Target-bidirectional Agreementcited by 40
  8. 2019Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translationcited by 40
  9. 2018Non-Autoregressive Neural Machine Translation with Enhanced Decoder Inputcited by 39
  10. 2019Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Informationcited by 37
  11. 2016Neural Machine Translation with Reconstructioncited by 28
  12. 2019Hint-Based Training for Non-Autoregressive Machine Translationcited by 28
  13. 2019Consistency by Agreement in Zero-shot Neural Machine Translationcited by 22
  14. 2019Syntactically Supervised Transformers for Faster Neural Machine Translationcited by 19
  15. 2020Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translationcited by 19
  16. 2019Non-autoregressive Transformer by Position Learningcited by 17
  17. 2019Retrieving Sequential Information for Non-Autoregressive Neural Machine Translationcited by 15
  18. 2017Modeling Past and Future for Neural Machine Translationcited by 6
  19. 2019Dynamic Past and Future for Neural Machine Translationcited by 5
  20. 2015Agreement-based Joint Training for Bidirectional Attention-based Neural Machine Translationcited by 4

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3.

4 papers.

  1. 2022One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translationcited by 20
  2. 2021Sequence-Level Training for Non-Autoregressive Neural Machine Translationcited by 18
  3. 2022Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translationcited by 16
  4. 2022Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 13

Similar, not cited

Close in wording, no citation either way

  1. 2020Aligned Cross Entropy for Non-Autoregressive Machine Translation
  2. 2022Helping the Weak Makes You Strong: Simple Multi-Task Learning Improves Non-Autoregressive Translators
  3. 2025Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation
  4. 2021Zero-Shot Translation using Diffusion Models
  5. 2024Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines
  6. 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping
  7. 2025Dual-objective Language Models: Training Efficiency Without Overfitting
  8. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time