mlsanityData to 14 Oct 2026

4D Panoptic Scene Graph Generation

arXiv 2405.10305 ↗First posted 16 May 2024cs.CV cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

We are living in a three-dimensional space while moving forward through a fourth dimension: time. To allow artificial intelligence to develop a comprehensive understanding of such a 4D environment, we introduce 4D Panoptic Scene Graph (PSG-4D), a new representation that bridges the raw visual data perceived in a dynamic 4D world and high-level visual understanding. Specifically, PSG-4D abstracts rich 4D sensory data into nodes, which represent entities with precise location and status information, and edges, which capture the temporal relations. To facilitate research in this new area, we build a richly annotated PSG-4D dataset consisting of 3K RGB-D videos with a total of 1M frames, each of which is labeled with 4D panoptic segmentation masks as well as fine-grained, dynamic scene graphs. To solve PSG-4D, we propose PSG4DFormer, a Transformer-based model that can predict panoptic segmentation masks, track masks along the time axis, and generate the corresponding scene graphs via a relation component. Extensive experiments on the new dataset show that our method can serve as a strong baseline for future research on PSG-4D. In the end, we provide a real-world application example to demonstrate how we can achieve dynamic scene understanding by integrating a large language model into our PSG-4D system.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 53 of its 70 references that are in the map. First posted on arXiv 16 May 2024.

Builds on

53 of 70 references.

References · 53

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Segment Anythingcited by 8,234
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  5. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  8. 2019Scalability in Perception for Autonomous Driving: Waymo Open Datasetcited by 2,451
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
  10. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  11. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  12. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
38 more
  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  2. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 748
  3. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  4. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 414
  5. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
  6. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  7. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 294
  8. 20193D Scene Graph: A Structure for Unified Semantics, 3D Space, and Cameracited by 281
  9. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 259
  10. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 250
  11. 2021Associating Objects with Transformers for Video Object Segmentationcited by 226
  12. 2020Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructionscited by 208
  13. 2023Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perceptioncited by 167
  14. 2021SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequencescited by 163
  15. 2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
  16. 2021EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devicescited by 139
  17. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  18. 2020Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentationcited by 122
  19. 2022Panoptic Scene Graph Generationcited by 114
  20. 2020GNN3DMOT: Graph Neural Network for 3D Multi-Object Tracking with Multi-Feature Learningcited by 113
  21. 2021Energy-Based Learning for Scene Graph Generationcited by 104
  22. 2021SGTR: End-to-end Scene Graph Generation with Transformercited by 103
  23. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 96
  24. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 96
  25. 2022Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentationcited by 79
  26. 20193-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agentscited by 74
  27. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  28. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
  29. 2021Do Different Tracking Tasks Require Different Appearance Models?cited by 62
  30. 2019Robust Multi-Modality Multi-Object Trackingcited by 56
  31. 20224D-OR: Semantic Scene Graphs for OR Domain Modelingcited by 49
  32. 2023Panoptic Video Scene Graph Generationcited by 46
  33. 2021SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction from Video Datacited by 38
  34. 20223D Instances as 1D Kernelscited by 36
  35. 2020Rel3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3Dcited by 30
  36. 2022Reasoning with Scene Graphs for Robot Planning under Partial Observabilitycited by 25
  37. 2023HiLo: Exploiting High Low Frequency Relations for Unbiased Panoptic Scene Graph Generationcited by 19
  38. 2023Pair then Relation: Pair-Net for Panoptic Scene Graph Generationcited by 17

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 14; 2026: 6.

22 papers.

Sort:

By area:

  1. 2025MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environmentscited by 28
  2. 2024CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videoscited by 11
  3. 2025Universal Scene Graph Generationcited by 11
  4. 2025How Do Images Align and Complement LiDAR? Towards a Harmonized Multi-modal 3D Panoptic Segmentationcited by 8
  5. 2025Digital Twin Generation from Visual Data: A Surveycited by 7
  6. 2025Unbiased Video Scene Graph Generation via Visual and Semantic Dual Debiasingcited by 6
  7. 2025Learning 4D Panoptic Scene Graph Generation from Rich 2D Visual Scenecited by 6
  8. 2025Agent Identity Evals: Measuring Agentic Identitycited by 5
  9. 2024A Fair Ranking and New Model for Panoptic Scene Graph Generationcited by 4
  10. 2025DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenescited by 4
  11. 2025Advances in 4D Representation: Geometry, Motion, and Interactioncited by 4
  12. 2026Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videoscited by 3
10 more
  1. 2025Hi-Dyna Graph: Hierarchical Dynamic Scene Graph for Robotic Autonomy in Human-Centric Environmentscited by 2
  2. 2025UNO: Unifying One-stage Video Scene Graph Generation via Object-Centric Visual Representation Learningcited by 2
  3. 2025FORESCENE: FOREcasting human activity via latent SCENE graphs diffusioncited by 1
  4. 2025SG-Tailor: Inter-Object Commonsense Relationship Reasoning for Scene Graph Manipulationcited by 1
  5. 2025M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environmentscited by 1
  6. 2026GTASA: Ground Truth Annotations for Spatiotemporal Analysis, Evaluation and Training of Video Modelscited by 0
  7. 2026Frequency-guided Multi-level Reasoning for Scene Graph Generation in Videocited by 0
  8. 2026PanDA: Unsupervised Domain Adaptation for Multimodal 3D Panoptic Segmentation in Autonomous Drivingcited by 0
  9. 2026PUF: Plug-and-Play Uncertainty-Aware Fusion for Online 3D Scene Graph Generationcited by 0
  10. 2026GESTO: Human-Centric Spatio-Temporal Memory for Reasoning in Dynamic Scenescited by 0