# Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation

arXiv: https://arxiv.org/abs/1906.09444 · HTML: https://mlsanity.com/paper/1906.09444/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2019.

## Built on by (most cited first, 15 of 15)

- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate](https://mlsanity.com/paper/2109.06481/) (2021)
- [One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation](https://mlsanity.com/paper/2205.14333/) (2022)
- [Sequence-Level Training for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2106.08122/) (2021)
- [O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning](https://mlsanity.com/paper/2108.02359/) (2021)
- [Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.03953/) (2022)
- [Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.05193/) (2022)
- [Guiding Teacher Forcing with Seer Forcing for Neural Machine Translation](https://mlsanity.com/paper/2106.06751/) (2021)
- [Language Generation with Strictly Proper Scoring Rules](https://mlsanity.com/paper/2405.18906/) (2024)
- [Beyond Next Token Prediction: Patch-Level Training for Large Language Models](https://mlsanity.com/paper/2407.12665/) (2024)
- [Multi-Granularity Optimization for Non-Autoregressive Translation](https://mlsanity.com/paper/2210.11017/) (2022)
- [Modeling Coverage for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2104.11897/) (2021)
- [Beyond MLE: Convex Learning for Text Generation](https://mlsanity.com/paper/2310.17217/) (2023)
- [Rephrasing the Reference for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2211.16863/) (2022)
- [What Have We Achieved on Non-autoregressive Translation?](https://mlsanity.com/paper/2405.12788/) (2024)

## Abstract (arXiv)

Non-Autoregressive Transformer (NAT) aims to accelerate the Transformer model through discarding the autoregressive mechanism and generating target words independently, which fails to exploit the target sequential information. Over-translation and under-translation errors often occur for the above reason, especially in the long sentence translation scenario. In this paper, we propose two approaches to retrieve the target sequential information for NAT to enhance its translation ability while preserving the fast-decoding property. Firstly, we propose a sequence-level training method based on a novel reinforcement algorithm for NAT (Reinforce-NAT) to reduce the variance and stabilize the training procedure. Secondly, we propose an innovative Transformer decoder named FS-decoder to fuse the target sequential information into the top layer of the decoder. Experimental results on three translation tasks show that the Reinforce-NAT surpasses the baseline NAT system by a significant margin on BLEU without decelerating the decoding speed and the FS-decoder achieves comparable translation performance to the autoregressive Transformer with considerable speedup.