mlsanityData to 7 Oct 2026

Panoptic Scene Graph Generation

arXiv 2207.11247 ↗First posted 22 Jul 2022cs.CV cs.AI cs.CLcited by 114

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing research addresses scene graph generation (SGG) -- a critical technology for scene understanding in images -- from a detection perspective, i.e., objects are detected using bounding boxes followed by prediction of their pairwise relationships. We argue that such a paradigm causes several problems that impede the progress of the field. For instance, bounding box-based labels in current datasets usually contain redundant classes like hairs, and leave out background information that is crucial to the understanding of context. In this work, we introduce panoptic scene graph generation (PSG), a new problem task that requires the model to generate a more comprehensive scene graph representation based on panoptic segmentations rather than rigid bounding boxes. A high-quality PSG dataset, which contains 49k well-annotated overlapping images from COCO and Visual Genome, is created for the community to keep track of its progress. For benchmarking, we build four two-stage baselines, which are modified from classic methods in SGG, and two one-stage baselines called PSGTR and PSGFormer, which are based on the efficient Transformer-based detector, i.e., DETR. While PSGTR uses a set of queries to directly learn triplets, PSGFormer separately models the objects and relations in the form of queries from two Transformer decoders, followed by a prompting-like relation-object matching mechanism. In the end, we share insights on open challenges and future directions.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Builds on 59 of its 78 references that are in the map. First posted on arXiv 22 Jul 2022.

Builds on

59 of 78 references.

References · 59

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2020End-to-End Object Detection with Transformerscited by 7,384
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  7. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  8. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  9. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  10. 2018Panoptic Segmentationcited by 796
  11. 2019Panoptic Feature Pyramid Networkscited by 644
  12. 2017Scene Graph Generation by Iterative Message Passingcited by 462
38 more
  1. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  2. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 355
  3. 2016Visual Relationship Detection with Language Priorscited by 333
  4. 2018Image Generation from Scene Graphscited by 324
  5. 2018Graph R-CNN for Scene Graph Generationcited by 296
  6. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 258
  7. 2017Learning to Detect Human-Object Interactionscited by 238
  8. 2021K-Net: Towards Unified Image Segmentationcited by 211
  9. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  10. 2015Visual Semantic Role Labelingcited by 188
  11. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  12. 2019UPSNet: A Unified Panoptic Segmentation Networkcited by 175
  13. 2020GPS-Net: Graph Property Sensing Network for Scene Graph Generationcited by 165
  14. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 163
  15. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  16. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 134
  17. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 129
  18. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  19. 2017Visual Translation Embedding Network for Visual Relation Detectioncited by 112
  20. 2020Learning Human-Object Interaction Detection using Interaction Pointscited by 108
  21. 2021Energy-Based Learning for Scene Graph Generationcited by 104
  22. 2018Explainable and Explicit Visual Reasoning over Scene Graphscited by 103
  23. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  24. 2019Scene Graph Generation with External Knowledge and Image Reconstructioncited by 102
  25. 2017Detecting Visual Relationships with Deep Relational Networkscited by 97
  26. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  27. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  28. 2020Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphscited by 87
  29. 2020Bridging Knowledge Graphs to Generate Scene Graphscited by 86
  30. 2020Semantic Image Manipulation Using Scene Graphscited by 82
  31. 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
  32. 2018Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generationcited by 78
  33. 2018Attentive Relational Networks for Mapping Images to Scene Graphscited by 72
  34. 2021Learning of Visual Relations: The Devil is in the Tailscited by 65
  35. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
  36. 2022The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generationcited by 60
  37. 2019Deep Contextual Attention for Human-Object Interaction Detectioncited by 54
  38. 2020Scene Graph Reasoning for Visual Question Answeringcited by 49

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 34; 2024: 29; 2025: 33; 2026: 17.

114 papers.

Sort:

By area:

  1. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 447
  2. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  3. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  4. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  5. 2024SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understandingcited by 101
  6. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  7. 2023ReVersion: Diffusion-Based Relation Inversion from Imagescited by 82
  8. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  9. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 68
  10. 2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
  11. 2023Going Denser with Open-Vocabulary Part Segmentationcited by 51
  12. 2023Action Scene Graphs for Long-Form Understanding of Egocentric Videoscited by 51
38 more
  1. 2023Mitigating Hallucination in Visual Language Models with Visual Supervisioncited by 49
  2. 2023Panoptic Video Scene Graph Generationcited by 46
  3. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Modelscited by 43
  4. 2023Panoptic Scene Graph Generation with Semantics-Prototype Learningcited by 42
  5. 2023You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videoscited by 41
  6. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  7. 2024MoAI: Mixture of All Intelligence for Large Language and Vision Modelscited by 36
  8. 2023Vision-Language Interpreter for Robot Task Planningcited by 29
  9. 2025MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environmentscited by 28
  10. 20244D Panoptic Scene Graph Generationcited by 22
  11. 2023Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captionscited by 21
  12. 2023HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understandingcited by 21
  13. 2024Learning Spatiotemporal Inconsistency via Thumbnail Layout for Face Deepfake Detectioncited by 21
  14. 2023HiLo: Exploiting High Low Frequency Relations for Unbiased Panoptic Scene Graph Generationcited by 19
  15. 2025Secure On-Device Video OOD Detection Without Backpropagationcited by 19
  16. 2025Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMscited by 18
  17. 2023Pair then Relation: Pair-Net for Panoptic Scene Graph Generationcited by 17
  18. 2024HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipationcited by 17
  19. 2024OpenPSG: Open-set Panoptic Scene Graph Generation via Large Multimodal Modelscited by 15
  20. 2023TextPSG: Panoptic Scene Graph Generation from Textual Descriptionscited by 14
  21. 2024LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relationscited by 13
  22. 2023VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generationcited by 12
  23. 2025COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generationcited by 12
  24. 2025R2-T2: Re-Routing in Test-Time for Multimodal Mixture-of-Expertscited by 12
  25. 2024DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generationcited by 11
  26. 2024CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videoscited by 11
  27. 2025Universal Scene Graph Generationcited by 11
  28. 2025Compile Scene Graphs with Reinforcement Learningcited by 11
  29. 2023Large Language Models as Automated Aligners for benchmarking Vision-Language Modelscited by 10
  30. 2026SAMTok: Representing Any Mask with Two Wordscited by 10
  31. 2023Haystack: A Panoptic Scene Graph Dataset to Evaluate Rare Predicate Classescited by 8
  32. 2024Leveraging Predicate and Triplet Learning for Scene Graph Generationcited by 8
  33. 2023Let There Be Order: Rethinking Ordering in Autoregressive Graph Generationcited by 7
  34. 2023Domain-wise Invariant Learning for Panoptic Scene Graph Generationcited by 7
  35. 2024Any2Graph: Deep End-To-End Supervised Graph Prediction With An Optimal Transport Losscited by 7
  36. 2025Open World Scene Graph Generation using Vision Language Modelscited by 7
  37. 2023AIMS: All-Inclusive Multi-Level Segmentationcited by 6
  38. 2025Learning 4D Panoptic Scene Graph Generation from Rich 2D Visual Scenecited by 6

50 most cited of 114 All 114 (JSON)