# Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translation

arXiv: https://arxiv.org/abs/2210.03953 · HTML: https://mlsanity.com/paper/2210.03953/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 8 Oct 2022.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Levenshtein Transformer](https://mlsanity.com/paper/1905.11006/) (2019)
- [End-to-end people detection in crowded scenes](https://mlsanity.com/paper/1506.04878/) (2015)
- [Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation](https://mlsanity.com/paper/2006.10369/) (2020)
- [Fast Decoding in Sequence Models using Discrete Latent Variables](https://mlsanity.com/paper/1803.03382/) (2018)
- [Glancing Transformer for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2008.07905/) (2020)
- [Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade](https://mlsanity.com/paper/2012.15833/) (2020)
- [Non-Autoregressive Machine Translation with Latent Alignments](https://mlsanity.com/paper/2004.07437/) (2020)
- [FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow](https://mlsanity.com/paper/1909.02480/) (2019)
- [End-to-End Non-Autoregressive Neural Machine Translation with Connectionist Temporal Classification](https://mlsanity.com/paper/1811.04719/) (2018)
- [Aligned Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2004.01655/) (2020)
- [Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior](https://mlsanity.com/paper/1908.07181/) (2019)
- [Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2106.05093/) (2021)
- [Theory and Experiments on Vector Quantized Autoencoders](https://mlsanity.com/paper/1805.11063/) (2018)
- [Non-Autoregressive Machine Translation with Auxiliary Regularization](https://mlsanity.com/paper/1902.10245/) (2019)
- [Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1911.09320/) (2019)
- [Non-Autoregressive Translation with Layer-Wise Prediction and Deep Supervision](https://mlsanity.com/paper/2110.07515/) (2021)
- [Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information](https://mlsanity.com/paper/1911.02215/) (2019)
- [Non-Autoregressive Translation by Learning Target Categorical Codes](https://mlsanity.com/paper/2103.11405/) (2021)
- [AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate](https://mlsanity.com/paper/2109.06481/) (2021)
- [One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation](https://mlsanity.com/paper/2205.14333/) (2022)
- [Sequence-Level Training for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2106.08122/) (2021)
- [Non-autoregressive Transformer by Position Learning](https://mlsanity.com/paper/1911.10677/) (2019)

## Built on by (most cited first, 16 of 16)

- [Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.05193/) (2022)
- [Non-autoregressive Streaming Transformer for Simultaneous Translation](https://mlsanity.com/paper/2310.14883/) (2023)
- [Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2303.06662/) (2023)
- [A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translation](https://mlsanity.com/paper/2406.06937/) (2024)
- [Continuous Autoregressive Language Models](https://mlsanity.com/paper/2510.27688/) (2025)
- [DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translation](https://mlsanity.com/paper/2310.07403/) (2023)
- [Beyond Next Token Prediction: Patch-Level Training for Large Language Models](https://mlsanity.com/paper/2407.12665/) (2024)
- [Beyond MLE: Convex Learning for Text Generation](https://mlsanity.com/paper/2310.17217/) (2023)
- [Overcoming Non-monotonicity in Transducer-based Streaming Generation](https://mlsanity.com/paper/2411.17170/) (2024)
- [Rephrasing the Reference for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2211.16863/) (2022)
- [N-Gram Nearest Neighbor Machine Translation](https://mlsanity.com/paper/2301.12866/) (2023)
- [Shared Latent Space by Both Languages in Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2305.03511/) (2023)
- [Non-Autoregressive Document-Level Machine Translation](https://mlsanity.com/paper/2305.12878/) (2023)
- [CTC-based Non-autoregressive Textless Speech-to-Speech Translation](https://mlsanity.com/paper/2406.07330/) (2024)
- [Improving Non-autoregressive Translation Quality with Pretrained Language Model, Embedding Distillation and Upsampling Strategy for CTC](https://mlsanity.com/paper/2306.06345/) (2023)
- [Transformer-Encoder Trees for Efficient Multilingual Machine Translation and Speech Translation](https://mlsanity.com/paper/2509.17930/) (2025)

## Abstract (arXiv)

Non-autoregressive translation (NAT) models are typically trained with the cross-entropy loss, which forces the model outputs to be aligned verbatim with the target sentence and will highly penalize small shifts in word positions. Latent alignment models relax the explicit alignment by marginalizing out all monotonic latent alignments with the CTC loss. However, they cannot handle non-monotonic alignments, which is non-negligible as there is typically global word reordering in machine translation. In this work, we explore non-monotonic latent alignments for NAT. We extend the alignment space to non-monotonic alignments to allow for the global word reordering and further consider all alignments that overlap with the target sentence. We non-monotonically match the alignments to the target sentence and train the latent alignment model to maximize the F1 score of non-monotonic matching. Extensive experiments on major WMT benchmarks show that our method substantially improves the translation performance of CTC-based models. Our best model achieves 30.06 BLEU on WMT14 En-De with only one-iteration decoding, closing the gap between non-autoregressive and autoregressive models.