# Comprehensive Image Captioning via Scene Graph Decomposition

arXiv: https://arxiv.org/abs/2007.11731 · HTML: https://mlsanity.com/paper/2007.11731/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2020.

## Built on by (most cited first, 30 of 52)

- [Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis](https://mlsanity.com/paper/2212.05032/) (2022)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning](https://mlsanity.com/paper/2110.01705/) (2021)
- [Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2203.09811/) (2022)
- [Learning to Generate Scene Graph from Natural Language Supervision](https://mlsanity.com/paper/2109.02227/) (2021)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [Instance Relation Graph Guided Source-Free Domain Adaptive Object Detection](https://mlsanity.com/paper/2203.15793/) (2022)
- [FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing](https://mlsanity.com/paper/2305.17497/) (2023)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [Human-like Controllable Image Captioning with Verb-specific Semantic Roles](https://mlsanity.com/paper/2103.12204/) (2021)
- [Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generation](https://mlsanity.com/paper/2103.05271/) (2021)
- [In Defense of Scene Graphs for Image Captioning](https://mlsanity.com/paper/2102.04990/) (2021)
- [LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation](https://mlsanity.com/paper/2310.10404/) (2023)
- [ReFormer: The Relational Transformer for Image Captioning](https://mlsanity.com/paper/2107.14178/) (2021)
- [Devil's on the Edges: Selective Quad Attention for Scene Graph Generation](https://mlsanity.com/paper/2304.03495/) (2023)
- [Resistance Training using Prior Bias: toward Unbiased Scene Graph Generation](https://mlsanity.com/paper/2201.06794/) (2022)
- [Contextual Modeling for 3D Dense Captioning on Point Clouds](https://mlsanity.com/paper/2210.03925/) (2022)
- [Pair then Relation: Pair-Net for Panoptic Scene Graph Generation](https://mlsanity.com/paper/2307.08699/) (2023)
- [Transforming Visual Scene Graphs to Image Captions](https://mlsanity.com/paper/2305.02177/) (2023)
- [Learning Combinatorial Prompts for Universal Controllable Image Captioning](https://mlsanity.com/paper/2303.06338/) (2023)
- [Graph Neural Networks in Vision-Language Image Understanding: A Survey](https://mlsanity.com/paper/2303.03761/) (2023)
- [Environment-Invariant Curriculum Relation Learning for Fine-Grained Scene Graph Generation](https://mlsanity.com/paper/2308.03282/) (2023)
- [Representing Videos as Discriminative Sub-graphs for Action Recognition](https://mlsanity.com/paper/2201.04027/) (2022)
- [Swinv2-Imagen: Hierarchical Vision Transformer Diffusion Models for Text-to-Image Generation](https://mlsanity.com/paper/2210.09549/) (2022)
- [VeriGraph: Scene Graphs for Execution Verifiable Robot Planning](https://mlsanity.com/paper/2411.10446/) (2024)
- [Visual Spatial Description: Controlled Spatial-Oriented Image-to-Text Generation](https://mlsanity.com/paper/2210.11109/) (2022)
- [ROOT: VLM based System for Indoor Scene Understanding and Beyond](https://mlsanity.com/paper/2411.15714/) (2024)
- [ESCA: Contextualizing Embodied Agents via Scene-Graph Generation](https://mlsanity.com/paper/2510.15963/) (2025)
- [Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting](https://mlsanity.com/paper/2310.18804/) (2023)

## Abstract (arXiv)

We address the challenging problem of image captioning by revisiting the representation of image scene graph. At the core of our method lies the decomposition of a scene graph into a set of sub-graphs, with each sub-graph capturing a semantic component of the input image. We design a deep model to select important sub-graphs, and to decode each selected sub-graph into a single target sentence. By using sub-graphs, our model is able to attend to different components of the image. Our method thus accounts for accurate, diverse, grounded and controllable captioning at the same time. We present extensive experiments to demonstrate the benefits of our comprehensive captioning model. Our method establishes new state-of-the-art results in caption diversity, grounding, and controllability, and compares favourably to latest methods in caption quality. Our project website can be found at http://pages.cs.wisc.edu/~yiwuzhong/Sub-GC.html.