# Semi-Autoregressive Training Improves Mask-Predict Decoding

arXiv: https://arxiv.org/abs/2001.08785 · HTML: https://mlsanity.com/paper/2001.08785/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2020.

## Built on by (most cited first, 30 of 33)

- [Continuous diffusion for categorical data](https://mlsanity.com/paper/2211.15089/) (2022)
- [A Reparameterized Discrete Diffusion Model for Text Generation](https://mlsanity.com/paper/2302.05737/) (2023)
- [Accelerating Transformer Inference for Translation via Parallel Decoding](https://mlsanity.com/paper/2305.10427/) (2023)
- [Step-unrolled Denoising Autoencoders for Text Generation](https://mlsanity.com/paper/2112.06749/) (2021)
- [Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems](https://mlsanity.com/paper/2312.15234/) (2023)
- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises](https://mlsanity.com/paper/2302.10025/) (2023)
- [Directed Acyclic Transformer for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2205.07459/) (2022)
- [Source and Target Bidirectional Knowledge Distillation for End-to-end Speech Translation](https://mlsanity.com/paper/2104.06457/) (2021)
- [Lossless Acceleration for Seq2seq Generation with Aggressive Decoding](https://mlsanity.com/paper/2205.10350/) (2022)
- [Non-Autoregressive Semantic Parsing for Compositional Task-Oriented Dialog](https://mlsanity.com/paper/2104.04923/) (2021)
- [Candidate Soups: Fusing Candidate Results Improves Translation Quality for Non-Autoregressive Translation](https://mlsanity.com/paper/2301.11503/) (2023)
- [Fuzzy Alignments in Directed Acyclic Graph for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2303.06662/) (2023)
- [Duplex Sequence-to-Sequence Learning for Reversible Machine Translation](https://mlsanity.com/paper/2105.03458/) (2021)
- [Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring](https://mlsanity.com/paper/2109.04411/) (2021)
- [Uncertainty-Aware Image Captioning](https://mlsanity.com/paper/2211.16769/) (2022)
- [Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines](https://mlsanity.com/paper/2407.21046/) (2024)
- [Towards Variable-Length Textual Adversarial Attacks](https://mlsanity.com/paper/2104.08139/) (2021)
- [MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation](https://mlsanity.com/paper/2108.08447/) (2021)
- [AMOM: Adaptive Masking over Masking for Conditional Masked Language Model](https://mlsanity.com/paper/2303.07457/) (2023)
- [Non-autoregressive Machine Translation with Probabilistic Context-free Grammar](https://mlsanity.com/paper/2311.07941/) (2023)
- [Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs](https://mlsanity.com/paper/2506.05387/) (2025)
- [Can Latent Alignments Improve Autoregressive Machine Translation?](https://mlsanity.com/paper/2104.09554/) (2021)
- [Improving Top-K Decoding for Non-Autoregressive Semantic Parsing via Intent Conditioning](https://mlsanity.com/paper/2204.06748/) (2022)
- [Hybrid-Regressive Neural Machine Translation](https://mlsanity.com/paper/2210.10416/) (2022)
- [Diffusion Glancing Transformer for Parallel Sequence to Sequence Learning](https://mlsanity.com/paper/2212.10240/) (2022)
- [Dynamic Alignment Mask CTC: Improved Mask-CTC with Aligned Cross Entropy](https://mlsanity.com/paper/2303.07687/) (2023)
- [Optimizing Non-Autoregressive Transformers with Contrastive Learning](https://mlsanity.com/paper/2305.13667/) (2023)
- [Encoder-decoder multimodal speaker change detection](https://mlsanity.com/paper/2306.00680/) (2023)
- [Emage: Non-Autoregressive Text-to-Image Generation](https://mlsanity.com/paper/2312.14988/) (2023)

## Abstract (arXiv)

The recently proposed mask-predict decoding algorithm has narrowed the performance gap between semi-autoregressive machine translation models and the traditional left-to-right approach. We introduce a new training method for conditional masked language models, SMART, which mimics the semi-autoregressive behavior of mask-predict, producing training examples that contain model predictions as part of their inputs. Models trained with SMART produce higher-quality translations when using mask-predict decoding, effectively closing the remaining performance gap with fully autoregressive models.