# Masked Non-Autoregressive Image Captioning

arXiv: https://arxiv.org/abs/1906.00717 · HTML: https://mlsanity.com/paper/1906.00717/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2019.

## Built on by (most cited first, 22 of 22)

- [A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond](https://mlsanity.com/paper/2204.09269/) (2022)
- [Semantic-Conditional Diffusion Networks for Image Captioning](https://mlsanity.com/paper/2212.03099/) (2022)
- [Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning](https://mlsanity.com/paper/2203.02573/) (2022)
- [M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers](https://mlsanity.com/paper/2105.14211/) (2021)
- [FlowTok: Flowing Seamlessly Across Text and Image Tokens](https://mlsanity.com/paper/2503.10772/) (2025)
- [Flowing from Words to Pixels: A Noise-Free Framework for Cross-Modality Evolution](https://mlsanity.com/paper/2412.15213/) (2024)
- [Automated Audio Captioning: An Overview of Recent Progress and New Challenges](https://mlsanity.com/paper/2205.05949/) (2022)
- [Exploring Discrete Diffusion Models for Image Captioning](https://mlsanity.com/paper/2211.11694/) (2022)
- [Semi-Autoregressive Transformer for Image Captioning](https://mlsanity.com/paper/2106.09436/) (2021)
- [CLIP-Diffusion-LM: Apply Diffusion Model on Image Captioning](https://mlsanity.com/paper/2210.04559/) (2022)
- [Semi-Autoregressive Image Captioning](https://mlsanity.com/paper/2110.05342/) (2021)
- [LaDiC: Are Diffusion Models Really Inferior to Autoregressive Counterparts for Image-to-Text Generation?](https://mlsanity.com/paper/2404.10763/) (2024)
- [Explicit Image Caption Editing](https://mlsanity.com/paper/2207.09625/) (2022)
- [Prefix-diffusion: A Lightweight Diffusion Model for Diverse Image Captioning](https://mlsanity.com/paper/2309.04965/) (2023)
- [Uncertainty-Aware Image Captioning](https://mlsanity.com/paper/2211.16769/) (2022)
- [Efficient Modeling of Future Context for Image Captioning](https://mlsanity.com/paper/2207.10897/) (2022)
- [DiffCap: Exploring Continuous Diffusion on Image Captioning](https://mlsanity.com/paper/2305.12144/) (2023)
- [DePA: Improving Non-autoregressive Machine Translation with Dependency-Aware Decoder](https://mlsanity.com/paper/2203.16266/) (2022)
- [Fast Sequence Generation with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2101.09698/) (2021)
- [Bounding and Filling: A Fast and Flexible Framework for Image Captioning](https://mlsanity.com/paper/2310.09876/) (2023)
- [Synergistic Dual Spatial-aware Generation of Image-to-Text and Text-to-Image](https://mlsanity.com/paper/2410.15312/) (2024)
- [Diff-3DCap: Shape Captioning with Diffusion Models](https://mlsanity.com/paper/2509.23718/) (2025)

## Abstract (arXiv)

Existing captioning models often adopt the encoder-decoder architecture, where the decoder uses autoregressive decoding to generate captions, such that each token is generated sequentially given the preceding generated tokens. However, autoregressive decoding results in issues such as sequential error accumulation, slow generation, improper semantics and lack of diversity. Non-autoregressive decoding has been proposed to tackle slow generation for neural machine translation but suffers from multimodality problem due to the indirect modeling of the target distribution. In this paper, we propose masked non-autoregressive decoding to tackle the issues of both autoregressive decoding and non-autoregressive decoding. In masked non-autoregressive decoding, we mask several kinds of ratios of the input sequences during training, and generate captions parallelly in several stages from a totally masked sequence to a totally non-masked sequence in a compositional manner during inference. Experimentally our proposed model can preserve semantic content more effectively and can generate more diverse captions.