# Improving Non-autoregressive Neural Machine Translation with Monolingual Data

arXiv: https://arxiv.org/abs/2005.00932 · HTML: https://mlsanity.com/paper/2005.00932/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2020.

## Built on by (most cited first, 10 of 10)

- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation](https://mlsanity.com/paper/2205.14333/) (2022)
- [Sequence-Level Training for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2106.08122/) (2021)
- [Enriching Non-Autoregressive Transformer with Syntactic and SemanticStructures for Neural Machine Translation](https://mlsanity.com/paper/2101.08942/) (2021)
- [Improving Simultaneous Machine Translation with Monolingual Data](https://mlsanity.com/paper/2212.01188/) (2022)
- [Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing](https://mlsanity.com/paper/2506.13485/) (2025)
- [TreePiece: Faster Semantic Parsing via Tree Tokenization](https://mlsanity.com/paper/2303.17161/) (2023)
- [KD4MT: A Survey of Knowledge Distillation for Machine Translation](https://mlsanity.com/paper/2602.15845/) (2026)
- [The JHU-Microsoft Submission for WMT21 Quality Estimation Shared Task](https://mlsanity.com/paper/2109.08724/) (2021)
- [Quick Back-Translation for Unsupervised Machine Translation](https://mlsanity.com/paper/2312.00912/) (2023)

## Abstract (arXiv)

Non-autoregressive (NAR) neural machine translation is usually done via knowledge distillation from an autoregressive (AR) model. Under this framework, we leverage large monolingual corpora to improve the NAR model's performance, with the goal of transferring the AR model's generalization ability while preventing overfitting. On top of a strong NAR baseline, our experimental results on the WMT14 En-De and WMT16 En-Ro news translation tasks confirm that monolingual data augmentation consistently improves the performance of the NAR model to approach the teacher AR model's performance, yields comparable or better results than the best non-iterative NAR methods in the literature and helps reduce overfitting in the training process.