# Semi-Autoregressive Transformer for Image Captioning

arXiv: https://arxiv.org/abs/2106.09436 · HTML: https://mlsanity.com/paper/2106.09436/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 37 references that are in the map.
- First posted on arXiv 17 Jun 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Unified Vision-Language Pre-Training for Image Captioning and VQA](https://mlsanity.com/paper/1909.11059/) (2019)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Attention on Attention for Image Captioning](https://mlsanity.com/paper/1908.06954/) (2019)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [X-Linear Attention Networks for Image Captioning](https://mlsanity.com/paper/2003.14080/) (2020)
- [Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement](https://mlsanity.com/paper/1802.06901/) (2018)
- [In Defense of Grid Features for Visual Question Answering](https://mlsanity.com/paper/2001.03615/) (2020)
- [Causal Attention for Vision-Language Tasks](https://mlsanity.com/paper/2103.03493/) (2021)
- [Non-Autoregressive Machine Translation with Auxiliary Regularization](https://mlsanity.com/paper/1902.10245/) (2019)
- [Semi-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1808.08583/) (2018)
- [More Grounded Image Captioning by Distilling Image-Text Matching Model](https://mlsanity.com/paper/2004.00390/) (2020)
- [Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input](https://mlsanity.com/paper/1812.09664/) (2018)
- [Non-Autoregressive Image Captioning with Counterfactuals-Critical Multi-Agent Learning](https://mlsanity.com/paper/2005.04690/) (2020)
- [A Better Variant of Self-Critical Sequence Training](https://mlsanity.com/paper/2003.09971/) (2020)
- [Masked Non-Autoregressive Image Captioning](https://mlsanity.com/paper/1906.00717/) (2019)
- [Fast Image Caption Generation with Position Alignment](https://mlsanity.com/paper/1912.06365/) (2019)

## Built on by (most cited first, 12 of 12)

- [Semantic-Conditional Diffusion Networks for Image Captioning](https://mlsanity.com/paper/2212.03099/) (2022)
- [FlowTok: Flowing Seamlessly Across Text and Image Tokens](https://mlsanity.com/paper/2503.10772/) (2025)
- [Flowing from Words to Pixels: A Noise-Free Framework for Cross-Modality Evolution](https://mlsanity.com/paper/2412.15213/) (2024)
- [Fine-grained Audible Video Description](https://mlsanity.com/paper/2303.15616/) (2023)
- [Uncertainty-Aware Image Captioning](https://mlsanity.com/paper/2211.16769/) (2022)
- [Semi-Autoregressive Streaming ASR With Label Context](https://mlsanity.com/paper/2309.10926/) (2023)
- [Image Captioning via Dynamic Path Customization](https://mlsanity.com/paper/2406.00334/) (2024)
- [SGDiff: Scene Graph Guided Diffusion Model for Image Collaborative SegCaptioning](https://mlsanity.com/paper/2512.01975/) (2025)
- [Bounding and Filling: A Fast and Flexible Framework for Image Captioning](https://mlsanity.com/paper/2310.09876/) (2023)
- [Progressive Neural Network for Multi-Horizon Time Series Forecasting](https://mlsanity.com/paper/2310.19322/) (2023)
- [ViTOC: Vision Transformer and Object-aware Captioner](https://mlsanity.com/paper/2411.07265/) (2024)
- [Diffusion Is Your Friend in Show, Suggest and Tell](https://mlsanity.com/paper/2512.10038/) (2025)

## Abstract (arXiv)

Current state-of-the-art image captioning models adopt autoregressive decoders, ie they generate each word by conditioning on previously generated words, which leads to heavy latency during inference. To tackle this issue, non-autoregressive image captioning models have recently been proposed to significantly accelerate the speed of inference by generating all words in parallel. However, these non-autoregressive models inevitably suffer from large generation quality degradation since they remove words dependence excessively. To make a better trade-off between speed and quality, we introduce a semi-autoregressive model for image captioning~(dubbed as SATIC), which keeps the autoregressive property in global but generates words parallelly in local . Based on Transformer, there are only a few modifications needed to implement SATIC. Experimental results on the MSCOCO image captioning benchmark show that SATIC can achieve a good trade-off without bells and whistles. Code is available at colormagentaurlhttps://github.com/YuanEZhou/satic.