mlsanityData to 14 Oct 2026

Panoptic Video Scene Graph Generation

arXiv 2311.17058 ↗First posted 28 Nov 2023cs.CV cs.AIcited by 46

Not on the map. Why?

Abstract (arXiv)

Towards building comprehensive real-world visual perception systems, we propose and study a new problem called panoptic scene graph generation (PVSG). PVSG relates to the existing video scene graph generation (VidSGG) problem, which focuses on temporal interactions between humans and objects grounded with bounding boxes in videos. However, the limitation of bounding boxes in detecting non-rigid objects and backgrounds often causes VidSGG to miss key details crucial for comprehensive video understanding. In contrast, PVSG requires nodes in scene graphs to be grounded by more precise, pixel-level segmentation masks, which facilitate holistic scene understanding. To advance research in this new area, we contribute the PVSG dataset, which consists of 400 videos (289 third-person + 111 egocentric videos) with a total of 150K frames labeled with panoptic segmentation masks as well as fine, temporal scene graphs. We also provide a variety of baseline methods and share useful design practices for future work.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 46 of its 62 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

46 of 62 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  8. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,103
  9. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  10. 2018Panoptic Segmentationcited by 802
  11. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  12. 2020Rescaling Egocentric Visioncited by 540
34 more
  1. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
  2. 2018TVQA: Localized, Compositional Video Question Answeringcited by 467
  3. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  4. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 414
  5. 2019Video Instance Segmentationcited by 393
  6. 2019Panoptic-DeepLabcited by 363
  7. 2017Temporal Relational Reasoning in Videoscited by 324
  8. 2018Graph R-CNN for Scene Graph Generationcited by 298
  9. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 259
  10. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 250
  11. 2015YFCC100M: The New Data in Multimedia Researchcited by 235
  12. 2021Associating Objects with Transformers for Video Object Segmentationcited by 226
  13. 2020Video Panoptic Segmentationcited by 142
  14. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 140
  15. 2022Panoptic Scene Graph Generationcited by 114
  16. 2021Energy-Based Learning for Scene Graph Generationcited by 104
  17. 2021SGTR: End-to-end Scene Graph Generation with Transformercited by 103
  18. 2020ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentationcited by 96
  19. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 96
  20. 2020Human-centric Spatio-Temporal Video Grounding With Visual Transformerscited by 85
  21. 2022Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentationcited by 79
  22. 2021STEP: Segmenting and Tracking Every Pixelcited by 73
  23. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
  24. 2021Do Different Tracking Tasks Require Different Appearance Models?cited by 62
  25. 2022The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generationcited by 60
  26. 2016Going Deeper into Action Recognition: A Surveycited by 55
  27. 2021Target Adaptive Context Aggregation for Video Scene Graph Generationcited by 52
  28. 2018Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systemscited by 42
  29. 2021PolyphonicFormer: Unified Query Learning for Depth-aware Video Panoptic Segmentationcited by 36
  30. 2022TubeFormer-DeepLab: Video Mask Transformercited by 35
  31. 2021Learning to Associate Every Segment for Video Panoptic Segmentationcited by 25
  32. 2022Meta Spatio-Temporal Debiasing for Video Scene Graph Generationcited by 20
  33. 2023Pair then Relation: Pair-Net for Panoptic Scene Graph Generationcited by 17
  34. 2022ZLPR: A Novel Loss for Multi-label Classificationcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 13; 2025: 18; 2026: 13.

46 papers.

Sort:

By area:

  1. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
  2. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 111
  3. 2025EgoLife: Towards Egocentric Life Assistantcited by 91
  4. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 75
  5. 2023Action Scene Graphs for Long-Form Understanding of Egocentric Videoscited by 51
  6. 2025MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environmentscited by 28
  7. 2025HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generationcited by 26
  8. 2023HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understandingcited by 22
  9. 20244D Panoptic Scene Graph Generationcited by 22
  10. 2024HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipationcited by 19
  11. 2024WorldQA: Multimodal World Knowledge in Videos through Long-Chain Reasoningcited by 16
  12. 2025LongViTU: Instruction Tuning for Long-Form Video Understandingcited by 13
34 more
  1. 2024CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videoscited by 11
  2. 2025Universal Scene Graph Generationcited by 11
  3. 2024Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generationcited by 10
  4. 2025RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graphcited by 7
  5. 2025Unbiased Video Scene Graph Generation via Visual and Semantic Dual Debiasingcited by 6
  6. 2025Learning 4D Panoptic Scene Graph Generation from Rich 2D Visual Scenecited by 6
  7. 2025From Vision To Language through Graph of Events in Space and Time: An Explainable Self-supervised Approachcited by 5
  8. 2026GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinkingcited by 5
  9. 2024Situational Scene Graph for Structured Human-centric Situation Understandingcited by 4
  10. 2025SAMJAM: Zero-Shot Video Scene Graph Generation for Egocentric Kitchen Videoscited by 4
  11. 2025DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenescited by 4
  12. 2025CAT-SG: A Large Dynamic Scene Graph Dataset for Fine-Grained Understanding of Cataract Surgerycited by 4
  13. 2024COM Kitchens: An Unedited Overhead-view Video Dataset as a Vision-Language Benchmarkcited by 3
  14. 2024Towards Flexible Visual Relationship Segmentationcited by 2
  15. 2025Hi-Dyna Graph: Hierarchical Dynamic Scene Graph for Robotic Autonomy in Human-Centric Environmentscited by 2
  16. 2025UNO: Unifying One-stage Video Scene Graph Generation via Object-Centric Visual Representation Learningcited by 2
  17. 2026MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representationcited by 2
  18. 2024Spatial-Temporal Multi-level Association for Video Object Segmentationcited by 1
  19. 2024Mapping the Unseen: Unified Promptable Panoptic Mapping with Dynamic Labeling using Foundation Modelscited by 1
  20. 2025FORESCENE: FOREcasting human activity via latent SCENE graphs diffusioncited by 1
  21. 2026SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Modelscited by 1
  22. 2026Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphscited by 1
  23. 2025THYME: Temporal Hierarchical-Cyclic Interactivity Modeling for Video Scene Graphs in Aerial Footagecited by 0
  24. 2025EventRR: Event Referential Reasoning for Referring Video Object Segmentationcited by 0
  25. 2025Click2Graph: Interactive Panoptic Video Scene Graphs from a Single Clickcited by 0
  26. 2026BUSSARD: Normalizing Flows for Bijective Universal Scene-Specific Anomalous Relationship Detectioncited by 0
  27. 2026GTASA: Ground Truth Annotations for Spatiotemporal Analysis, Evaluation and Training of Video Modelscited by 0
  28. 2026CAGE-SGG: Counterfactual Active Graph Evidence for Open-Vocabulary Scene Graph Generationcited by 0
  29. 2026ReLIC-SGG: Relation Lattice Completion for Open-Vocabulary Scene Graph Generationcited by 0
  30. 2026One Identity, Many Roles: Multimodal Entity Coreference for Enhanced Video Situation Recognitioncited by 0
  31. 2026BARISTA: A Multi-Task Egocentric Benchmark for Compositional Visual Understandingcited by 0
  32. 2026T-STAR: A Large-Scale Benchmark for Spatio-Temporal Panoptic Scene Graph Generation in Satellite Videocited by 0
  33. 2026Semantic Action Graph: A Shared Representation for Agent Grounding and Human Interpretation of Sports Highlightscited by 0
  34. 2026Panoptic Scene Program Diffusion Transformercited by 0