# Revisiting Non-Autoregressive Translation at Scale

arXiv: https://arxiv.org/abs/2305.16155 · HTML: https://mlsanity.com/paper/2305.16155/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 25 May 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019)
- [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://mlsanity.com/paper/1907.05019/) (2019)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [Understanding Knowledge Distillation in Non-autoregressive Machine Translation](https://mlsanity.com/paper/1911.02727/) (2019)
- [Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation](https://mlsanity.com/paper/2006.10369/) (2020)
- [Glancing Transformer for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2008.07905/) (2020)
- [Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade](https://mlsanity.com/paper/2012.15833/) (2020)
- [Non-Autoregressive Machine Translation with Latent Alignments](https://mlsanity.com/paper/2004.07437/) (2020)
- [Document-Level Machine Translation with Large Language Models](https://mlsanity.com/paper/2304.02210/) (2023)
- [Aligned Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2004.01655/) (2020)
- [Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2106.05093/) (2021)
- [Understanding and Improving Lexical Choice in Non-Autoregressive Translation](https://mlsanity.com/paper/2012.14583/) (2020)
- [Non-Autoregressive Machine Translation with Auxiliary Regularization](https://mlsanity.com/paper/1902.10245/) (2019)
- [compare-mt: A Tool for Holistic Comparison of Language Generation Systems](https://mlsanity.com/paper/1903.07926/) (2019)
- [Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1911.09320/) (2019)
- [Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation](https://mlsanity.com/paper/2106.00903/) (2021)
- [ParroT: Translating during Chat using Large Language Models tuned with Human Translation and Feedback](https://mlsanity.com/paper/2304.02426/) (2023)
- [Context-Aware Cross-Attention for Non-Autoregressive Translation](https://mlsanity.com/paper/2011.00770/) (2020)
- [Progressive Multi-Granularity Training for Non-Autoregressive Translation](https://mlsanity.com/paper/2106.05546/) (2021)
- [A Paradigm Shift: The Future of Machine Translation Lies with Large Language Models](https://mlsanity.com/paper/2305.01181/) (2023)
- [Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2010.12868/) (2020)

## Built on by (most cited first, 2 of 2)

- [Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs](https://mlsanity.com/paper/2311.03127/) (2023)
- [On the Information Redundancy in Non-Autoregressive Translation](https://mlsanity.com/paper/2405.02673/) (2024)

## Abstract (arXiv)

In real-world systems, scaling has been critical for improving the translation quality in autoregressive translation (AT), which however has not been well studied for non-autoregressive translation (NAT). In this work, we bridge the gap by systematically studying the impact of scaling on NAT behaviors. Extensive experiments on six WMT benchmarks over two advanced NAT models show that scaling can alleviate the commonly-cited weaknesses of NAT models, resulting in better translation performance. To reduce the side-effect of scaling on decoding speed, we empirically investigate the impact of NAT encoder and decoder on the translation performance. Experimental results on the large-scale WMT20 En-De show that the asymmetric architecture (e.g. bigger encoder and smaller decoder) can achieve comparable performance with the scaling model, while maintaining the superiority of decoding speed with standard NAT models. To this end, we establish a new benchmark by validating scaled NAT models on the scaled dataset, which can be regarded as a strong baseline for future works. We release code and system outputs at https://github.com/DeepLearnXMU/Scaling4NAT.