# Skeleton Key: Image Captioning by Skeleton-Attribute Decomposition

arXiv: https://arxiv.org/abs/1704.06972 · HTML: https://mlsanity.com/paper/1704.06972/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2017.

## Built on by (most cited first, 15 of 15)

- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning](https://mlsanity.com/paper/2102.10407/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptions](https://mlsanity.com/paper/2105.04489/) (2021)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [Cross-Modal Causal Intervention for Medical Report Generation](https://mlsanity.com/paper/2303.09117/) (2023)
- [The Role of Syntactic Planning in Compositional Image Captioning](https://mlsanity.com/paper/2101.11911/) (2021)
- [Prophet Attention: Predicting Attention with Future Attention for Image Captioning](https://mlsanity.com/paper/2210.10914/) (2022)
- [Reverse Stable Diffusion: What prompt was used to generate this image?](https://mlsanity.com/paper/2308.01472/) (2023)
- [Exploring Semantic Relationships for Unpaired Image Captioning](https://mlsanity.com/paper/2106.10658/) (2021)
- [CLID: Controlled-Length Image Descriptions with Limited Data](https://mlsanity.com/paper/2211.14835/) (2022)
- [Goal-driven text descriptions for images](https://mlsanity.com/paper/2108.12575/) (2021)
- [Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis](https://mlsanity.com/paper/2408.04909/) (2024)
- [RefineCap: Concept-Aware Refinement for Image Captioning](https://mlsanity.com/paper/2109.03529/) (2021)

## Abstract (arXiv)

Recently, there has been a lot of interest in automatically generating descriptions for an image. Most existing language-model based approaches for this task learn to generate an image description word by word in its original word order. However, for humans, it is more natural to locate the objects and their relationships first, and then elaborate on each object, describing notable attributes. We present a coarse-to-fine method that decomposes the original image description into a skeleton sentence and its attributes, and generates the skeleton sentence and attribute phrases separately. By this decomposition, our method can generate more accurate and novel descriptions than the previous state-of-the-art. Experimental results on the MS-COCO and a larger scale Stock3M datasets show that our algorithm yields consistent improvements across different evaluation metrics, especially on the SPICE metric, which has much higher correlation with human ratings than the conventional metrics. Furthermore, our algorithm can generate descriptions with varied length, benefiting from the separate control of the skeleton and attributes. This enables image description generation that better accommodates user preferences.