# AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate

arXiv: https://arxiv.org/abs/2109.06481 · HTML: https://mlsanity.com/paper/2109.06481/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 14 Sep 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [Understanding Knowledge Distillation in Non-autoregressive Machine Translation](https://mlsanity.com/paper/1911.02727/) (2019)
- [Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation](https://mlsanity.com/paper/2006.10369/) (2020)
- [Glancing Transformer for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2008.07905/) (2020)
- [Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade](https://mlsanity.com/paper/2012.15833/) (2020)
- [Non-Autoregressive Machine Translation with Latent Alignments](https://mlsanity.com/paper/2004.07437/) (2020)
- [FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow](https://mlsanity.com/paper/1909.02480/) (2019)
- [End-to-End Adversarial Text-to-Speech](https://mlsanity.com/paper/2006.03575/) (2020)
- [Aligned Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2004.01655/) (2020)
- [Fast Structured Decoding for Sequence Models](https://mlsanity.com/paper/1910.11555/) (2019)
- [Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior](https://mlsanity.com/paper/1908.07181/) (2019)
- [Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1911.09320/) (2019)
- [Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information](https://mlsanity.com/paper/1911.02215/) (2019)
- [An EM Approach to Non-autoregressive Conditional Sequence Generation](https://mlsanity.com/paper/2006.16378/) (2020)
- [Syntactically Supervised Transformers for Faster Neural Machine Translation](https://mlsanity.com/paper/1906.02780/) (2019)
- [On The Alignment Problem In Multi-Head Attention-Based Neural Machine Translation](https://mlsanity.com/paper/1809.03985/) (2018)
- [Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1906.09444/) (2019)
- [Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2009.07177/) (2020)
- [Enriching Non-Autoregressive Transformer with Syntactic and SemanticStructures for Neural Machine Translation](https://mlsanity.com/paper/2101.08942/) (2021)

## Built on by (most cited first, 24 of 24)

- [Accelerating Transformer Inference for Translation via Parallel Decoding](https://mlsanity.com/paper/2305.10427/) (2023)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models](https://mlsanity.com/paper/2508.08712/) (2025)
- [One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation](https://mlsanity.com/paper/2205.14333/) (2022)
- [Lossless Acceleration for Seq2seq Generation with Aggressive Decoding](https://mlsanity.com/paper/2205.10350/) (2022)
- [Reducing Position Bias in Simultaneous Machine Translation with Length-Aware Framework](https://mlsanity.com/paper/2203.09053/) (2022)
- [Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.03953/) (2022)
- [Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.05193/) (2022)
- [Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translation](https://mlsanity.com/paper/2301.11503/) (2023)
- [CTC-based Non-autoregressive Speech Translation](https://mlsanity.com/paper/2305.17358/) (2023)
- [Multi-Granularity Optimization for Non-Autoregressive Translation](https://mlsanity.com/paper/2210.11017/) (2022)
- [Non-Autoregressive Neural Machine Translation: A Call for Clarity](https://mlsanity.com/paper/2205.10577/) (2022)
- [Non-autoregressive Machine Translation with Probabilistic Context-free Grammar](https://mlsanity.com/paper/2311.07941/) (2023)
- [Curriculum Learning for Biological Sequence Prediction: The Case of De Novo Peptide Sequencing](https://mlsanity.com/paper/2506.13485/) (2025)
- [Rephrasing the Reference for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2211.16863/) (2022)
- [Diffusion Glancing Transformer for Parallel Sequence to Sequence Learning](https://mlsanity.com/paper/2212.10240/) (2022)
- [Plan-then-Seam: Towards Efficient Table-to-Text Generation](https://mlsanity.com/paper/2302.05138/) (2023)
- [A Major Obstacle for NLP Research: Let's Talk about Time Allocation!](https://mlsanity.com/paper/2211.16858/) (2022)
- [RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformer](https://mlsanity.com/paper/2303.07665/) (2023)
- [EM-Network: Oracle Guided Self-distillation for Sequence Learning](https://mlsanity.com/paper/2306.10058/) (2023)
- [Which Tokens Need Context? A Reference-Based Analysis of Translation Responsibility Using Fertility and Entropy](https://mlsanity.com/paper/2606.29489/) (2026)
- [DiMS: Distilling Multiple Steps of Iterative Non-Autoregressive Transformers for Machine Translation](https://mlsanity.com/paper/2206.02999/) (2022)
- [Improving Non-autoregressive Translation Quality with Pretrained Language Model, Embedding Distillation and Upsampling Strategy for CTC](https://mlsanity.com/paper/2306.06345/) (2023)
- [Improving Non-autoregressive Machine Translation with Error Exposure and Consistency Regularization](https://mlsanity.com/paper/2402.09725/) (2024)

## Abstract (arXiv)

Non-autoregressive neural machine translation (NART) models suffer from the multi-modality problem which causes translation inconsistency such as token repetition. Most recent approaches have attempted to solve this problem by implicitly modeling dependencies between outputs. In this paper, we introduce AligNART, which leverages full alignment information to explicitly reduce the modality of the target distribution. AligNART divides the machine translation task into (i) alignment estimation and (ii) translation with aligned decoder inputs, guiding the decoder to focus on simplified one-to-one translation. To alleviate the alignment estimation problem, we further propose a novel alignment decomposition method. Our experiments show that AligNART outperforms previous non-iterative NART models that focus on explicit modality reduction on WMT14 EnleftrightarrowDe and WMT16 RorightarrowEn. Furthermore, AligNART achieves BLEU scores comparable to those of the state-of-the-art connectionist temporal classification based models on WMT14 EnleftrightarrowDe. We also observe that AligNART effectively addresses the token repetition problem even without sequence-level knowledge distillation.