mlsanityData to 14 Oct 2026

Graph Neural Networks in Vision-Language Image Understanding: A Survey

arXiv 2303.03761 ↗First posted 7 Mar 2023cs.CV cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

2D image understanding is a complex problem within computer vision, but it holds the key to providing human-level scene comprehension. It goes further than identifying the objects in an image, and instead, it attempts to understand the scene. Solutions to this problem form the underpinning of a range of tasks, including image captioning, visual question answering (VQA), and image retrieval. Graphs provide a natural way to represent the relational arrangement between objects in an image, and thus, in recent years graph neural networks (GNNs) have become a standard component of many 2D image understanding pipelines, becoming a core architectural component, especially in the VQA group of tasks. In this survey, we review this rapidly evolving field and we provide a taxonomy of graph types used in 2D image understanding approaches, a comprehensive list of the GNN models used in this domain, and a roadmap of future potential developments. To the best of our knowledge, this is the first comprehensive survey that covers image captioning, visual question answering, and image retrieval techniques that focus on using GNNs as the main part of their architecture.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 87 of its 165 references that are in the map. First posted on arXiv 7 Mar 2023.

Builds on

87 of 165 references.

References · 87

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  7. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  9. 2017Graph Attention Networkscited by 8,923
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  11. 2017Inductive Representation Learning on Large Graphscited by 6,320
  12. 2018How Powerful are Graph Neural Networks?cited by 4,393
38 more
  1. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  2. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  5. 2015VQA: Visual Question Answeringcited by 2,506
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  7. 2019Towards VQA Models That Can Readcited by 1,718
  8. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
  9. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  12. 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 881
  13. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  14. 2015Gated Graph Sequence Neural Networkscited by 796
  15. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  16. 2020Knowledge Graphscited by 621
  17. 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 545
  18. 2019MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixingcited by 443
  19. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  20. 2020Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloudcited by 335
  21. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 306
  22. 2018Graph R-CNN for Scene Graph Generationcited by 298
  23. 2021GRAND: Graph Neural Diffusioncited by 258
  24. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 255
  25. 2021Weisfeiler and Lehman Go Cellular: CW Networkscited by 218
  26. 2018Exploring Visual Relationship for Image Captioningcited by 215
  27. 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 214
  28. 2022Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNscited by 214
  29. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  30. 2016FVQA: Fact-based Visual Question Answeringcited by 181
  31. 2019Image Captioning: Transforming Objects into Wordscited by 171
  32. 2020When Radiology Report Generation Meets Knowledge Graphcited by 145
  33. 2017Automatic Spatially-aware Fashion Concept Discoverycited by 134
  34. 2021A Comprehensive Survey of Scene Graphs: Generation and Applicationcited by 125
  35. 2016Graph-Structured Representations for Visual Question Answeringcited by 123
  36. 2019Relation-Aware Graph Attention Network for Visual Question Answeringcited by 119
  37. 2021SOON: Scenario Oriented Object Navigation with Graph-based Explorationcited by 119
  38. 2022Understanding and Extending Subgraph GNNs by Rethinking Their Symmetriescited by 112

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 2; 2025: 5.

10 papers.

  1. 2024Enhancing Visual Question Answering through Ranking-Based Hybrid Training and Multimodal Fusioncited by 16
  2. 2023Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluationcited by 12
  3. 2024A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenescited by 10
  4. 2023A Survey on Image-text Multimodal Modelscited by 4
  5. 2025MAS-KCL: Knowledge component graph structure learning with large language model-based agentic workflowcited by 3
  6. 2025Attention-based transformer models for image captioning across languages: An in-depth survey and evaluationcited by 1
  7. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 1
  8. 2023MUSTANG: Multi-Stain Self-Attention Graph Multiple Instance Learning Pipeline for Histopathology Whole Slide Imagescited by 0
  9. 2025SuperCap: Multi-resolution Superpixel-based Image Captioningcited by 0
  10. 2025Why Relational Graphs Will Save the Next Generation of Vision Foundation Models?cited by 0