mlsanityData to 7 Oct 2026

Segmentation-grounded Scene Graph Generation

arXiv 2104.14207 ↗First posted 29 Apr 2021cs.CVcited by 25

Not on the map. Why?

Abstract (arXiv)

Scene graph generation has emerged as an important problem in computer vision. While scene graphs provide a grounded representation of objects, their locations and relations in an image, they do so only at the granularity of proposal bounding boxes. In this work, we propose the first, to our knowledge, framework for pixel-level segmentation-grounded scene graph generation. Our framework is agnostic to the underlying scene graph generation method and address the lack of segmentation annotations in target scene graph datasets (e.g., Visual Genome) through transfer and multi-task learning from, and with, an auxiliary dataset (e.g., MS COCO). Specifically, each target object being detected is endowed with a segmentation mask, which is expressed as a lingual-similarity weighted linear combination over categories that have annotations present in an auxiliary dataset. These inferred masks, along with a novel Gaussian attention mechanism which grounds the relations at a pixel-level within the image, allow for improved relation prediction. The entire framework is end-to-end trainable and is learned in a multi-task manner with both target and auxiliary datasets.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 38 of its 58 references that are in the map. First posted on arXiv 29 Apr 2021.

Builds on

38 of 58 references.

References · 38

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2017Mask R-CNNcited by 7,259
  6. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  9. 2017A Survey on Multi-Task Learningcited by 838
  10. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  11. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  12. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
26 more
  1. 2016Visual Relationship Detection with Language Priorscited by 333
  2. 2018Recent Advances in Open Set Recognition: A Surveycited by 309
  3. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  4. 2015What's the Point: Semantic Segmentation with Point Supervisioncited by 297
  5. 2018Graph R-CNN for Scene Graph Generationcited by 296
  6. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 258
  7. 2016Segmentation from Natural Language Expressionscited by 257
  8. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 254
  9. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 254
  10. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  11. 2019Zero-Shot Semantic Segmentationcited by 245
  12. 2020GPS-Net: Graph Property Sensing Network for Scene Graph Generationcited by 165
  13. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 163
  14. 2018Video Object Segmentation with Language Referring Expressionscited by 161
  15. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  16. 2021Energy-Based Learning for Scene Graph Generationcited by 104
  17. 2020Bridging Knowledge Graphs to Generate Scene Graphscited by 86
  18. 2019Unpaired Image Captioning via Scene Graph Alignmentscited by 82
  19. 2018Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generationcited by 78
  20. 2017Open Vocabulary Scene Parsingcited by 76
  21. 2018Attentive Relational Networks for Mapping Images to Scene Graphscited by 72
  22. 2017Learning to Segment Every Thingcited by 64
  23. 2017Pixels to Graphs by Associative Embeddingcited by 58
  24. 2020Learning Visual Commonsense for Robust Scene Graph Generationcited by 47
  25. 2020Graph Density-Aware Losses for Novel Compositions in Scene Graph Generationcited by 31
  26. 2015Convex Learning of Multiple Tasks and their Structurecited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 10; 2023: 7; 2024: 2; 2025: 3; 2026: 2.

25 papers.

Sort:

By area:

  1. 2022Panoptic Scene Graph Generationcited by 114
  2. 2021SGTR: End-to-end Scene Graph Generation with Transformercited by 102
  3. 2022Latent Graph Representations for Critical View of Safety Assessmentcited by 37
  4. 2022Iterative Scene Graph Generationcited by 24
  5. 2023Devil's on the Edges: Selective Quad Attention for Scene Graph Generationcited by 23
  6. 2023Unbiased Scene Graph Generation via Two-stage Causal Modelingcited by 21
  7. 2022Collaborative Transformers for Grounded Situation Recognitioncited by 19
  8. 2023HiLo: Exploiting High Low Frequency Relations for Unbiased Panoptic Scene Graph Generationcited by 19
  9. 2022Hierarchical Memory Learning for Fine-Grained Scene Graph Generationcited by 15
  10. 2022Adaptive Fine-Grained Predicates Learning for Scene Graph Generationcited by 14
  11. 2023TextPSG: Panoptic Scene Graph Generation from Textual Descriptionscited by 14
  12. 2025A Causal Adjustment Module for Debiasing Scene Graph Generationcited by 14
13 more
  1. 2023DSG: An End-to-End Document Structure Generatorcited by 6
  2. 2022Explore Contextual Information for 3D Scene Graph Generationcited by 4
  3. 2022Peer Learning for Unbiased Scene Graph Generationcited by 4
  4. 2023HAtt-Flow: Hierarchical Attention-Flow Mechanism for Group Activity Scene Graph Generation in Videoscited by 4
  5. 2024Voxel Scene Graph for Intracranial Hemorrhagecited by 4
  6. 2022Context-aware Mixture-of-Experts for Unbiased Scene Graph Generationcited by 3
  7. 2022Scalable Modular Synthetic Data Generation for Advancing Aerial Autonomycited by 1
  8. 2024SGTR+: End-to-end Scene Graph Generation with Transformercited by 1
  9. 2025A Reverse Causal Framework to Mitigate Spurious Correlations for Debiasing Scene Graph Generationcited by 1
  10. 2026Sterilizable Scene Graph Generation for Operating Roomscited by 1
  11. 2023Self-Supervised Relation Alignment for Scene Graph Generationcited by 0
  12. 2025THYME: Temporal Hierarchical-Cyclic Interactivity Modeling for Video Scene Graphs in Aerial Footagecited by 0
  13. 2026Modular Diffusion Models for Structured Visual Recognitioncited by 0