# Enriching Non-Autoregressive Transformer with Syntactic and SemanticStructures for Neural Machine Translation

arXiv: https://arxiv.org/abs/2101.08942 · HTML: https://mlsanity.com/paper/2101.08942/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 22 Jan 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Scaling Neural Machine Translation](https://mlsanity.com/paper/1806.00187/) (2018)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [An Actor-Critic Algorithm for Sequence Prediction](https://mlsanity.com/paper/1607.07086/) (2016)
- [Insertion Transformer: Flexible Sequence Generation via Insertion Operations](https://mlsanity.com/paper/1902.03249/) (2019)
- [Fast Decoding in Sequence Models using Discrete Latent Variables](https://mlsanity.com/paper/1803.03382/) (2018)
- [Non-Autoregressive Machine Translation with Latent Alignments](https://mlsanity.com/paper/2004.07437/) (2020)
- [FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow](https://mlsanity.com/paper/1909.02480/) (2019)
- [End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification](https://mlsanity.com/paper/1811.04719/) (2018)
- [Aligned Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2004.01655/) (2020)
- [Fast Structured Decoding for Sequence Models](https://mlsanity.com/paper/1910.11555/) (2019)
- [Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior](https://mlsanity.com/paper/1908.07181/) (2019)
- [Non-Autoregressive Machine Translation with Auxiliary Regularization](https://mlsanity.com/paper/1902.10245/) (2019)
- [Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1911.09320/) (2019)
- [Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input](https://mlsanity.com/paper/1812.09664/) (2018)
- [Imitation Learning for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1906.02041/) (2019)
- [Hint-Based Training for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/1909.06708/) (2019)
- [KERMIT: Generative Insertion-Based Modeling for Sequences](https://mlsanity.com/paper/1906.01604/) (2019)
- [Improving Non-autoregressive Neural Machine Translation with Monolingual Data](https://mlsanity.com/paper/2005.00932/) (2020)
- [LAVA NAT: A Non-Autoregressive Translation Model with Look-Around Decoding and Vocabulary Attention](https://mlsanity.com/paper/2002.03084/) (2020)

## Built on by (most cited first, 13 of 13)

- [Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer](https://mlsanity.com/paper/2108.06625/) (2021)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models](https://mlsanity.com/paper/2508.08712/) (2025)
- [AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate](https://mlsanity.com/paper/2109.06481/) (2021)
- [Modeling Sequences as Distributions with Uncertainty for Sequential Recommendation](https://mlsanity.com/paper/2106.06165/) (2021)
- [One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation](https://mlsanity.com/paper/2205.14333/) (2022)
- [HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization](https://mlsanity.com/paper/2110.06388/) (2021)
- [Localize, Retrieve and Fuse: A Generalized Framework for Free-Form Question Answering over Tables](https://mlsanity.com/paper/2309.11049/) (2023)
- [Non-autoregressive Machine Translation with Probabilistic Context-free Grammar](https://mlsanity.com/paper/2311.07941/) (2023)
- [Rephrasing the Reference for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2211.16863/) (2022)
- [RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformer](https://mlsanity.com/paper/2303.07665/) (2023)
- [NARRepair: Non-Autoregressive Code Generation Model for Automatic Program Repair](https://mlsanity.com/paper/2406.16526/) (2024)
- [Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matching](https://mlsanity.com/paper/2604.15009/) (2026)

## Abstract (arXiv)

The non-autoregressive models have boosted the efficiency of neural machine translation through parallelized decoding at the cost of effectiveness when comparing with the autoregressive counterparts. In this paper, we claim that the syntactic and semantic structures among natural language are critical for non-autoregressive machine translation and can further improve the performance. However, these structures are rarely considered in the existing non-autoregressive models. Inspired by this intuition, we propose to incorporate the explicit syntactic and semantic structures of languages into a non-autoregressive Transformer, for the task of neural machine translation. Moreover, we also consider the intermediate latent alignment within target sentences to better learn the long-term token dependencies. Experimental results on two real-world datasets (i.e., WMT14 En-De and WMT16 En-Ro) show that our model achieves a significantly faster speed, as well as keeps the translation quality when compared with several state-of-the-art non-autoregressive models.