# Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine Translation

arXiv: https://arxiv.org/abs/2010.12868 · HTML: https://mlsanity.com/paper/2010.12868/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 16 of 16)

- [Accelerating Transformer Inference for Translation via Parallel Decoding](https://mlsanity.com/paper/2305.10427/) (2023)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervision](https://mlsanity.com/paper/2110.07515/) (2021)
- [Rejuvenating Low-Frequency Words: Making the Most of Parallel Data in Non-Autoregressive Translation](https://mlsanity.com/paper/2106.00903/) (2021)
- [O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning](https://mlsanity.com/paper/2108.02359/) (2021)
- [Lossless Acceleration for Seq2seq Generation with Aggressive Decoding](https://mlsanity.com/paper/2205.10350/) (2022)
- [Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translation](https://mlsanity.com/paper/2301.11503/) (2023)
- [Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring](https://mlsanity.com/paper/2109.04411/) (2021)
- [Compact and Optimal Deep Learning with Recurrent Parameter Generators](https://mlsanity.com/paper/2107.07110/) (2021)
- [Non-Autoregressive Neural Machine Translation: A Call for Clarity](https://mlsanity.com/paper/2205.10577/) (2022)
- [AMOM: Adaptive Masking over Masking for Conditional Masked Language Model](https://mlsanity.com/paper/2303.07457/) (2023)
- [Revisiting Non-Autoregressive Translation at Scale](https://mlsanity.com/paper/2305.16155/) (2023)
- [Helping the Weak Makes You Strong: Simple Multi-Task Learning Improves Non-Autoregressive Translators](https://mlsanity.com/paper/2211.06075/) (2022)
- [RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformer](https://mlsanity.com/paper/2303.07665/) (2023)
- [Improving Non-autoregressive Machine Translation with Error Exposure and Consistency Regularization](https://mlsanity.com/paper/2402.09725/) (2024)
- [Leveraging Diverse Modeling Contexts with Collaborating Learning for Neural Machine Translation](https://mlsanity.com/paper/2402.18428/) (2024)

## Abstract (arXiv)

Non-Autoregressive machine Translation (NAT) models have demonstrated significant inference speedup but suffer from inferior translation accuracy. The common practice to tackle the problem is transferring the Autoregressive machine Translation (AT) knowledge to NAT models, e.g., with knowledge distillation. In this work, we hypothesize and empirically verify that AT and NAT encoders capture different linguistic properties of source sentences. Therefore, we propose to adopt Multi-Task learning to transfer the AT knowledge to NAT models through encoder sharing. Specifically, we take the AT model as an auxiliary task to enhance NAT model performance. Experimental results on WMT14 English-German and WMT16 English-Romanian datasets show that the proposed Multi-Task NAT achieves significant improvements over the baseline NAT models. Furthermore, the performance on large-scale WMT19 and WMT20 English-German datasets confirm the consistency of our proposed method. In addition, experimental results demonstrate that our Multi-Task NAT is complementary to knowledge distillation, the standard knowledge transfer method for NAT.