mlsanityData to 7 Oct 2026

Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade

arXiv 2012.15833 ↗First posted 31 Dec 2020cs.CLcited by 99

Not on the map. Why?

Abstract (arXiv)

Fully non-autoregressive neural machine translation (NAT) is proposed to simultaneously predict tokens with single forward of neural networks, which significantly reduces the inference latency at the expense of quality drop compared to the Transformer baseline. In this work, we target on closing the performance gap while maintaining the latency advantage. We first inspect the fundamental issues of fully NAT models, and adopt dependency reduction in the learning space of output tokens as the basic guidance. Then, we revisit methods in four different aspects that have been proven effective for improving NAT models, and carefully combine these techniques with necessary modifications. Our extensive experiments on three translation benchmarks show that the proposed system achieves the new state-of-the-art results for fully NAT models, and obtains comparable performance with the autoregressive and iterative NAT systems. For instance, one of the proposed models achieves 27.49 BLEU points on WMT14 En-De with approximately 16.5X speed up at inference time.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 34; 2023: 22; 2024: 15; 2025: 8; 2026: 7.

99 papers.

Sort:

By area:

  1. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  2. 2022MAGVIT: Masked Generative Video Transformercited by 341
  3. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  4. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  5. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  6. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  7. 2024Energy-Based Diffusion Language Models for Text Generationcited by 86
  8. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  9. 2022Visual Prompt Tuning for Generative Transfer Learningcited by 75
  10. 2021BLT: Bidirectional Layout Transformer for Controllable Layout Generationcited by 69
  11. 2022A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyondcited by 64
  12. 2022Improved Masked Image Generation with Token-Criticcited by 63
38 more
  1. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decodingcited by 53
  2. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noisescited by 50
  3. 2022Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 44
  4. 2021Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervisioncited by 40
  5. 2022Empowering Diffusion Models on the Embedding Space for Text Generationcited by 40
  6. 2022On the Learning of Non-Autoregressive Transformerscited by 27
  7. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  8. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  9. 2022CTC Alignments Improve Autoregressive Translationcited by 25
  10. 2021AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translatecited by 24
  11. 2024Revisiting Non-Autoregressive Transformers for Efficient Image Synthesiscited by 22
  12. 2022One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translationcited by 20
  13. 2021Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correctioncited by 19
  14. 2022Lossless Acceleration for Seq2seq Generation with Aggressive Decodingcited by 18
  15. 2022Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translationcited by 16
  16. 2024SurfPro: Functional Protein Design Based on Continuous Surfacecited by 14
  17. 2022Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 13
  18. 2023Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translationcited by 13
  19. 2024BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Modelscited by 13
  20. 2024AdaNAT: Exploring Adaptive Policy for Token-Based Image Generationcited by 13
  21. 2022Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarizationcited by 12
  22. 2022Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence taskscited by 12
  23. 2022Non-Autoregressive Machine Translation: It's Not as Fast as it Seemscited by 11
  24. 2023Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translationcited by 11
  25. 2021Duplex Sequence-to-Sequence Learning for Reversible Machine Translationcited by 9
  26. 2021Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoringcited by 9
  27. 2022LegoNN: Building Modular Encoder-Decoder Modelscited by 9
  28. 2022SoftCorrect: Error Correction with Soft Detection for Automatic Speech Recognitioncited by 9
  29. 2023CTC-based Non-autoregressive Speech Translationcited by 9
  30. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9
  31. 2025Continuous Autoregressive Language Modelscited by 9
  32. 2021Self-Distillation Mixup Training for Non-autoregressive Neural Machine Translationcited by 8
  33. 2022ngram-OAXE: Phrase-Based Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translationcited by 8
  34. 2023Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generationcited by 8
  35. 2023Non-autoregressive Text Editing with Copy-aware Latent Alignmentscited by 8
  36. 2024The Mystery of the Pathological Path-star Task for Language Modelscited by 8
  37. 2024ENAT: Rethinking Spatial-temporal Interactions in Token-based Image Synthesiscited by 8
  38. 2021Improving Non-autoregressive Generation with Mixup Trainingcited by 7

50 most cited of 99 All 99 (JSON)