Data to 7 Oct 2026

Exploring Visual Relationship for Image Captioning

arXiv 1809.07041 ↗First posted 19 Sep 2018cs.CVcited by 215

Not on the map. Why?

Abstract (arXiv)

It is always well believed that modeling relationships between objects would be helpful for representing and eventually describing an image. Nevertheless, there has not been evidence in support of the idea on image description generation. In this paper, we introduce a new design to explore the connections between objects for image captioning under the umbrella of attention-based encoder-decoder framework. Specifically, we present Graph Convolutional Networks plus Long Short-Term Memory (dubbed as GCN-LSTM) architecture that novelly integrates both semantic and spatial object relationships into image encoder. Technically, we build graphs over the detected objects in an image based on their spatial and semantic connections. The representations of each region proposed on objects are then refined by leveraging graph structure through GCN. With the learnt region-level features, our GCN-LSTM capitalizes on LSTM-based captioning framework with attention mechanism for sentence generation. Extensive experiments are conducted on COCO image captioning dataset, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, GCN-LSTM increases CIDEr-D performance from 120.1% to 128.7% on COCO testing set.

Key facts. Cited by 215 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 79; 2022: 57; 2023: 34; 2024: 22; 2025: 17; 2026: 6.

215 papers.

Sort:

By area:

2021ClipCap: CLIP Prefix for Image Captioningcited by 557
2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 137
2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
2021Causal Attention for Vision-Language Taskscited by 111
2023DetCLIPv2: Scalable Open-Vocabulary Object Detection Pre-training via Word-Region Alignmentcited by 90
2021Dual-Level Collaborative Transformer for Image Captioningcited by 71
2021Recovering the Unbiased Scene Graphs from the Biased Onescited by 71
2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70

38 more

2021Learning of Visual Relations: The Devil is in the Tailscited by 65
2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
2021Fully Convolutional Scene Graph Generationcited by 57
2022GRIT: Faster and Better Image captioning Transformer Using Dual Visual Featurescited by 56
2022A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspectivecited by 56
2021CPTR: Full Transformer Network for Image Captioningcited by 55
2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 54
2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
2022Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detectioncited by 46
2021Structured Sparse R-CNN for Direct Scene Graph Generationcited by 44
2023RLIPv2: Fast Scaling of Relational Language-Image Pre-trainingcited by 44
2022Deep Learning Approaches on Image Captioning: A Reviewcited by 43
2022Modeling Image Composition for Complex Scene Generationcited by 43
2021Context-Aware Layout to Image Generation with Enhanced Object Appearancecited by 41
2021Unifying Multimodal Transformer for Bi-directional Image and Text Generationcited by 41
2021Injecting Semantic Concepts into End-to-End Image Captioningcited by 41
2023Expert Knowledge-Aware Image Difference Graph Representation Learning for Difference-Aware Medical Visual Question Answeringcited by 41
2024An Image Is Worth 1000 Lies: Adversarial Transferability across Prompts on Vision-Language Modelscited by 40
2022Retrieval-Augmented Transformer for Image Captioningcited by 38
2022Semantic-Conditional Diffusion Networks for Image Captioningcited by 38
2022End-to-End Transformer Based Model for Image Captioningcited by 37
2023Relational Context Learning for Human-Object Interaction Detectioncited by 35
2024MeaCap: Memory-Augmented Zero-shot Image Captioningcited by 35
2022Comprehending and Ordering Semantics for Image Captioningcited by 33
2023ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishingcited by 33
2021In Defense of Scene Graphs for Image Captioningcited by 31
2021Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graphcited by 31
2021ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integrationcited by 31
2021Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphscited by 30
2024Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Modelscited by 30
2021A Survey on Temporal Sentence Grounding in Videoscited by 29
2021Consensus Graph Representation Learning for Better Grounded Image Captioningcited by 28
2023Retrieval-augmented Image Captioningcited by 28
2024OV-DINO: Unified Open-Vocabulary Detection with Language-Aware Selective Fusioncited by 28
2024EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detectioncited by 28
2021Dual Graph Convolutional Networks with Transformer and Curriculum Learning for Image Captioningcited by 27

50 most cited of 215 All 215 (JSON)