# Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input

arXiv: https://arxiv.org/abs/1812.09664 · HTML: https://mlsanity.com/paper/1812.09664/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Dec 2018.

## Built on by (most cited first, 30 of 39)

- [Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems](https://mlsanity.com/paper/2312.15234/) (2023)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [Gradient-Based Constrained Sampling from Language Models](https://mlsanity.com/paper/2205.12558/) (2022)
- [A Study on ReLU and Softmax in Transformer](https://mlsanity.com/paper/2302.06461/) (2023)
- [Directed Acyclic Transformer for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2205.07459/) (2022)
- [FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognition](https://mlsanity.com/paper/2105.03842/) (2021)
- [Empowering Diffusion Models on the Embedding Space for Text Generation](https://mlsanity.com/paper/2212.09412/) (2022)
- [Non-Autoregressive Text Generation with Pre-trained Language Models](https://mlsanity.com/paper/2102.08220/) (2021)
- [M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers](https://mlsanity.com/paper/2105.14211/) (2021)
- [Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT](https://mlsanity.com/paper/2102.07594/) (2021)
- [FastCorrect 2: Fast Error Correction on Multiple Candidates for Automatic Speech Recognition](https://mlsanity.com/paper/2109.14420/) (2021)
- [Non-Autoregressive Translation by Learning Target Categorical Codes](https://mlsanity.com/paper/2103.11405/) (2021)
- [Sequence-Level Training for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2106.08122/) (2021)
- [O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning](https://mlsanity.com/paper/2108.02359/) (2021)
- [Recent Advances in Neural Text Generation: A Task-Agnostic Survey](https://mlsanity.com/paper/2203.03047/) (2022)
- [Enriching Non-Autoregressive Transformer with Syntactic and SemanticStructures for Neural Machine Translation](https://mlsanity.com/paper/2101.08942/) (2021)
- [Semi-Autoregressive Transformer for Image Captioning](https://mlsanity.com/paper/2106.09436/) (2021)
- [ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generation](https://mlsanity.com/paper/2210.13304/) (2022)
- [StraIT: Non-autoregressive Generation with Stratified Image Transformer](https://mlsanity.com/paper/2303.00750/) (2023)
- [Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring](https://mlsanity.com/paper/2109.04411/) (2021)
- [CTC-based Non-autoregressive Speech Translation](https://mlsanity.com/paper/2305.17358/) (2023)
- [NAST: Non-Autoregressive Spatial-Temporal Transformer for Time Series Forecasting](https://mlsanity.com/paper/2102.05624/) (2021)
- [FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis](https://mlsanity.com/paper/2207.03800/) (2022)
- [ngram-OAXE: Phrase-Based Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.03999/) (2022)
- [Improving Non-autoregressive Generation with Mixup Training](https://mlsanity.com/paper/2110.11115/) (2021)
- [Towards Variable-Length Textual Adversarial Attacks](https://mlsanity.com/paper/2104.08139/) (2021)
- [A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models](https://mlsanity.com/paper/2405.13019/) (2024)
- [Sentence-Permuted Paragraph Generation](https://mlsanity.com/paper/2104.07228/) (2021)
- [Modeling Coverage for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2104.11897/) (2021)
- [Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing](https://mlsanity.com/paper/2506.13485/) (2025)

## Abstract (arXiv)

Non-autoregressive translation (NAT) models, which remove the dependence on previous target tokens from the inputs of the decoder, achieve significantly inference speedup but at the cost of inferior accuracy compared to autoregressive translation (AT) models. Previous work shows that the quality of the inputs of the decoder is important and largely impacts the model accuracy. In this paper, we propose two methods to enhance the decoder inputs so as to improve NAT models. The first one directly leverages a phrase table generated by conventional SMT approaches to translate source tokens to target tokens, which are then fed into the decoder as inputs. The second one transforms source-side word embeddings to target-side word embeddings through sentence-level alignment and word-level adversary learning, and then feeds the transformed word embeddings into the decoder as inputs. Experimental results show our method largely outperforms the NAT baseline~citepgu2017non by 5.11 BLEU scores on WMT14 English-German task and 4.72 BLEU scores on WMT16 English-Romanian task.