mlsanityData to 7 Oct 2026

A Comprehensive Survey of Scene Graphs: Generation and Application

arXiv 2104.01111 ↗First posted 17 Mar 2021cs.CVcited by 125

Not on the map. Why?

Abstract (arXiv)

Scene graph is a structured representation of a scene that can clearly express the objects, attributes, and relationships between objects in the scene. As computer vision technology continues to develop, people are no longer satisfied with simply detecting and recognizing objects in images; instead, people look forward to a higher level of understanding and reasoning about visual scenes. For example, given an image, we want to not only detect and recognize objects in the image, but also know the relationship between objects (visual relationship detection), and generate a text description (image captioning) based on the image content. Alternatively, we might want the machine to tell us what the little girl in the image is doing (Visual Question Answering (VQA)), or even remove the dog from the image and find similar images (image editing and retrieval), etc. These tasks require a higher level of understanding and reasoning for image vision tasks. The scene graph is just such a powerful tool for scene understanding. Therefore, scene graphs have attracted the attention of a large number of researchers, and related research is often cross-modal, complex, and rapidly developing. However, no relatively systematic survey of scene graphs exists at present. To this end, this survey conducts a comprehensive investigation of the current scene graph research. More specifically, we first summarized the general definition of the scene graph, then conducted a comprehensive and systematic discussion on the generation method of the scene graph (SGG) and the SGG with the aid of prior knowledge. We then investigated the main applications of scene graphs and summarized the most commonly used datasets. Finally, we provide some insights into the future development of scene graphs. We believe this will be a very helpful foundation for future research on scene graphs.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Builds on 128 of its 189 references that are in the map. First posted on arXiv 17 Mar 2021.

Builds on

128 of 189 references.

References · 128

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  7. 2017Mask R-CNNcited by 7,259
  8. 2017Inductive Representation Learning on Large Graphscited by 6,284
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  10. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  11. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  12. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
38 more
  1. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  3. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  4. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,085
  5. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  6. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  7. 2015Gated Graph Sequence Neural Networkscited by 792
  8. 2015Holistically-Nested Edge Detectioncited by 590
  9. 2016Neural Architectures for Named Entity Recognitioncited by 548
  10. 2020A Survey of Deep Active Learningcited by 547
  11. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  12. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  13. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  14. 2017A simple neural network module for relational reasoningcited by 345
  15. 2014Recurrent Models of Visual Attentioncited by 344
  16. 2016Visual Relationship Detection with Language Priorscited by 333
  17. 2018Image Generation from Scene Graphscited by 324
  18. 2016Learning Convolutional Neural Networks for Graphscited by 314
  19. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  20. 2018Graph R-CNN for Scene Graph Generationcited by 296
  21. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 280
  22. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 258
  23. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 254
  24. 2020A Comprehensive Survey of Neural Architecture Search: Challenges and Solutionscited by 253
  25. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  26. 2017Learning to Detect Human-Object Interactionscited by 238
  27. 2018Videos as Space-Time Region Graphscited by 221
  28. 2015Conditional Random Fields as Recurrent Neural Networkscited by 213
  29. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 205
  30. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  31. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  32. 2020GPS-Net: Graph Property Sensing Network for Scene Graph Generationcited by 165
  33. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 163
  34. 2018Zero-shot Recognition via Semantic Embeddings and Knowledge Graphscited by 155
  35. 2017Scene Graph Generation from Objects, Phrases and Region Captionscited by 154
  36. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  37. 2016Harnessing Deep Neural Networks with Logic Rulescited by 135
  38. 2016Discriminative Embeddings of Latent Variable Models for Structured Datacited by 130

50 most cited of 128 All 128 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 15; 2023: 28; 2024: 32; 2025: 28; 2026: 20.

125 papers.

Sort:

By area:

  1. 2022Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communicationscited by 166
  2. 2022Scene Graph Generation: A Comprehensive Surveycited by 73
  3. 2024EchoScene: Indoor Scene Generation via Information Echo over Scene Graph Diffusioncited by 66
  4. 2023Graph-based Topology Reasoning for Driving Scenescited by 53
  5. 2023Panoptic Scene Graph Generation with Semantics-Prototype Learningcited by 42
  6. 2023Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open Worldcited by 35
  7. 2025Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?cited by 33
  8. 2024Planetarium: A Rigorous Benchmark for Translating Text to Structured Planning Languagescited by 32
  9. 2022VAEL: Bridging Variational Autoencoders and Probabilistic Logic Programmingcited by 31
  10. 2024STAR: A First-Ever Dataset and A Large-Scale Benchmark for Scene Graph Generation in Large-Size Satellite Imagerycited by 31
  11. 2024Benchmarking Large Vision-Language Models via Directed Scene Graph for Comprehensive Image Captioningcited by 30
  12. 2025RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaborationcited by 28
38 more
  1. 2023SG-Bot: Object Rearrangement via Coarse-to-Fine Robotic Imagination on Scene Graphscited by 24
  2. 2025Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunitiescited by 23
  3. 2023Unbiased Scene Graph Generation via Two-stage Causal Modelingcited by 21
  4. 2024Continual Learning on Graphs: Challenges, Solutions, and Opportunitiescited by 20
  5. 2022NICEST: Noisy Label Correction and Training for Robust Scene Graph Generationcited by 19
  6. 2023Pair then Relation: Pair-Net for Panoptic Scene Graph Generationcited by 17
  7. 2022RoMFAC: A robust mean-field actor-critic reinforcement learning against adversarial perturbations on statescited by 16
  8. 2023Learning Situation Hyper-Graphs for Video Question Answeringcited by 15
  9. 2021From SLAM to Situational Awareness: Challenges and Surveycited by 14
  10. 2022Label Semantic Knowledge Distillation for Unbiased Scene Graph Generationcited by 14
  11. 2024HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generationcited by 14
  12. 2025A Causal Adjustment Module for Debiasing Scene Graph Generationcited by 14
  13. 2022Rethinking Few-Shot Class-Incremental Learning with Open-Set Hypothesis in Hyperbolic Geometrycited by 12
  14. 2023Graph Neural Networks in Vision-Language Image Understanding: A Surveycited by 10
  15. 2023Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Trainingcited by 10
  16. 2023Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generationcited by 10
  17. 2024Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generationcited by 10
  18. 2023Generative modeling, design and analysis of spider silk protein sequences for enhanced mechanical propertiescited by 9
  19. 2025Controllable 3D Outdoor Scene Generation via Scene Graphscited by 9
  20. 2025LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Studycited by 9
  21. 2023Haystack: A Panoptic Scene Graph Dataset to Evaluate Rare Predicate Classescited by 8
  22. 2025A Neurosymbolic Framework for Interpretable Cognitive Attack Detection in Augmented Realitycited by 8
  23. 2022Swinv2-Imagen: Hierarchical Vision Transformer Diffusion Models for Text-to-Image Generationcited by 7
  24. 2023Domain-wise Invariant Learning for Panoptic Scene Graph Generationcited by 7
  25. 2023Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledgecited by 7
  26. 2024Any2Graph: Deep End-To-End Supervised Graph Prediction With An Optimal Transport Losscited by 7
  27. 2023Weakly-Supervised Learning of Visual Relations in Multimodal Pretrainingcited by 6
  28. 2023DSG: An End-to-End Document Structure Generatorcited by 6
  29. 2024Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoningcited by 6
  30. 2024Tri-modal Confluence with Temporal Dynamics for Scene Graph Generation in Operating Roomscited by 6
  31. 2021Single Image Dehazing with An Independent Detail-Recovery Networkcited by 5
  32. 2022Large Scale Time-Series Representation Learning via Simultaneous Low and High Frequency Feature Bootstrappingcited by 5
  33. 2023Synthesizing Event-centric Knowledge Graphs of Daily Activities Using Virtual Spacecited by 5
  34. 2024AUG: A New Dataset and An Efficient Model for Aerial Image Urban Scene Graph Generationcited by 5
  35. 2024Semantic Similarity Score for Measuring Visual Similarity at Semantic Levelcited by 5
  36. 2025PRISM-0: A Predicate-Rich Scene Graph Generation Framework for Zero-Shot Open-Vocabulary Taskscited by 5
  37. 2022Beyond Instance Discrimination: Relation-aware Contrastive Self-supervised Learningcited by 4
  38. 2023Improving Scene Graph Generation with Superpixel-Based Interaction Learningcited by 4

50 most cited of 125 All 125 (JSON)