# An EM Approach to Non-autoregressive Conditional Sequence Generation

arXiv: https://arxiv.org/abs/2006.16378 · HTML: https://mlsanity.com/paper/2006.16378/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2020.

## Built on by (most cited first, 21 of 21)

- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2106.05093/) (2021)
- [Directed Acyclic Transformer for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2205.07459/) (2022)
- [On the Learning of Non-Autoregressive Transformers](https://mlsanity.com/paper/2206.05975/) (2022)
- [AligNART: Non-autoregressive Neural Machine Translation by Jointly Learning to Estimate Alignment and Translate](https://mlsanity.com/paper/2109.06481/) (2021)
- [textitlatent-GLAT: Glancing at Latent Variables for Parallel Text Generation](https://mlsanity.com/paper/2204.02030/) (2022)
- [One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation](https://mlsanity.com/paper/2205.14333/) (2022)
- [Sequence-Level Training for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2106.08122/) (2021)
- [VigDet: Knowledge Informed Neural Temporal Point Process for Coordination Detection on Social Media](https://mlsanity.com/paper/2110.15454/) (2021)
- [Viterbi Decoding of Directed Acyclic Transformer for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.05193/) (2022)
- [Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring](https://mlsanity.com/paper/2109.04411/) (2021)
- [Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models](https://mlsanity.com/paper/2505.03075/) (2025)
- [ngram-OAXE: Phrase-Based Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2210.03999/) (2022)
- [Multi-Granularity Optimization for Non-Autoregressive Translation](https://mlsanity.com/paper/2210.11017/) (2022)
- [Non-Autoregressive Neural Machine Translation: A Call for Clarity](https://mlsanity.com/paper/2205.10577/) (2022)
- [Rephrasing the Reference for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2211.16863/) (2022)
- [Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/2303.17910/) (2023)
- [Optimizing Non-Autoregressive Transformers with Contrastive Learning](https://mlsanity.com/paper/2305.13667/) (2023)
- [A Monte Carlo Framework for Calibrated Uncertainty Estimation in Sequence Prediction](https://mlsanity.com/paper/2410.23272/) (2024)
- [DiMS: Distilling Multiple Steps of Iterative Non-Autoregressive Transformers for Machine Translation](https://mlsanity.com/paper/2206.02999/) (2022)
- [FlashTrie: A GPU-Accelerated Constrained Beam Search for Generative Retrieval](https://mlsanity.com/paper/2607.10044/) (2026)

## Abstract (arXiv)

Autoregressive (AR) models have been the dominating approach to conditional sequence generation, but are suffering from the issue of high inference latency. Non-autoregressive (NAR) models have been recently proposed to reduce the latency by generating all output tokens in parallel but could only achieve inferior accuracy compared to their autoregressive counterparts, primarily due to a difficulty in dealing with the multi-modality in sequence generation. This paper proposes a new approach that jointly optimizes both AR and NAR models in a unified Expectation-Maximization (EM) framework. In the E-step, an AR model learns to approximate the regularized posterior of the NAR model. In the M-step, the NAR model is updated on the new posterior and selects the training examples for the next AR model. This iterative process can effectively guide the system to remove the multi-modality in the output sequences. To our knowledge, this is the first EM approach to NAR sequence generation. We evaluate our method on the task of machine translation. Experimental results on benchmark data sets show that the proposed approach achieves competitive, if not better, performance with existing NAR models and significantly reduces the inference latency.