# Fast, Diverse and Accurate Image Captioning Guided By Part-of-Speech

arXiv: https://arxiv.org/abs/1805.12589 · HTML: https://mlsanity.com/paper/1805.12589/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2018.

## Built on by (most cited first, 30 of 55)

- [DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models](https://mlsanity.com/paper/2210.08933/) (2022)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [TESS: Text-to-Text Self-Conditioned Simplex Diffusion](https://mlsanity.com/paper/2305.08379/) (2023)
- [SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers](https://mlsanity.com/paper/2212.10325/) (2022)
- [CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models](https://mlsanity.com/paper/2302.04012/) (2023)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [LayoutDiffusion: Improving Graphic Layout Generation by Discrete Diffusion Probabilistic Models](https://mlsanity.com/paper/2303.11589/) (2023)
- [Empowering Diffusion Models on the Embedding Space for Text Generation](https://mlsanity.com/paper/2212.09412/) (2022)
- [ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing](https://mlsanity.com/paper/2303.02437/) (2023)
- [SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text](https://mlsanity.com/paper/2204.11964/) (2022)
- [Consensus Graph Representation Learning for Better Grounded Image Captioning](https://mlsanity.com/paper/2112.00974/) (2021)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Towards Accurate Text-based Image Captioning with Content Diversity Exploration](https://mlsanity.com/paper/2105.03236/) (2021)
- [Detecting Subtle Differences between Human and Model Languages Using Spectrum of Relative Likelihood](https://mlsanity.com/paper/2406.19874/) (2024)
- [IC3: Image Captioning by Committee Consensus](https://mlsanity.com/paper/2302.01328/) (2023)
- [Contextual Modeling for 3D Dense Captioning on Point Clouds](https://mlsanity.com/paper/2210.03925/) (2022)
- [Generating Visual Spatial Description via Holistic 3D Scene Understanding](https://mlsanity.com/paper/2305.11768/) (2023)
- [Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors](https://mlsanity.com/paper/2204.01227/) (2022)
- [The Role of Syntactic Planning in Compositional Image Captioning](https://mlsanity.com/paper/2101.11911/) (2021)
- [Learning Distinct and Representative Styles for Image Captioning](https://mlsanity.com/paper/2209.08231/) (2022)
- [Learning Combinatorial Prompts for Universal Controllable Image Captioning](https://mlsanity.com/paper/2303.06338/) (2023)
- [Parts of Speech-Grounded Subspaces in Vision-Language Models](https://mlsanity.com/paper/2305.14053/) (2023)
- [Question-controlled Text-aware Image Captioning](https://mlsanity.com/paper/2108.02059/) (2021)
- [Object-Centric Unsupervised Image Captioning](https://mlsanity.com/paper/2112.00969/) (2021)
- [Controllable Video Captioning with an Exemplar Sentence](https://mlsanity.com/paper/2112.01073/) (2021)
- [LSD-StructureNet: Modeling Levels of Structural Detail in 3D Part Hierarchies](https://mlsanity.com/paper/2108.13459/) (2021)
- [Smoothie: Smoothing Diffusion on Token Embeddings for Text Generation](https://mlsanity.com/paper/2505.18853/) (2025)
- [Image Captioning via Compact Bidirectional Architecture](https://mlsanity.com/paper/2201.01984/) (2022)
- [On Distinctive Image Captioning via Comparing and Reweighting](https://mlsanity.com/paper/2204.03938/) (2022)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)

## Abstract (arXiv)

Image captioning is an ambiguous problem, with many suitable captions for an image. To address ambiguity, beam search is the de facto method for sampling multiple captions. However, beam search is computationally expensive and known to produce generic captions. To address this concern, some variational auto-encoder (VAE) and generative adversarial net (GAN) based methods have been proposed. Though diverse, GAN and VAE are less accurate. In this paper, we first predict a meaningful summary of the image, then generate the caption based on that summary. We use part-of-speech as summaries, since our summary should drive caption generation. We achieve the trifecta: (1) High accuracy for the diverse captions as evaluated by standard captioning metrics and user studies; (2) Faster computation of diverse captions compared to beam search and diverse beam search; and (3) High diversity as evaluated by counting novel sentences, distinct n-grams and mutual overlap (i.e., mBleu-4) scores.