mlsanityData to 7 Oct 2026

Towards Scene Graph Anticipation

arXiv 2403.04899 ↗First posted 7 Mar 2024cs.CV cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Spatio-temporal scene graphs represent interactions in a video by decomposing scenes into individual objects and their pair-wise temporal relationships. Long-term anticipation of the fine-grained pair-wise relationships between objects is a challenging problem. To this end, we introduce the task of Scene Graph Anticipation (SGA). We adapt state-of-the-art scene graph generation methods as baselines to anticipate future pair-wise relationships between objects and propose a novel approach SceneSayer. In SceneSayer, we leverage object-centric representations of relationships to reason about the observed video frames and model the evolution of relationships between objects. We take a continuous time perspective and model the latent dynamics of the evolution of object interactions using concepts of NeuralODE and NeuralSDE, respectively. We infer representations of future relationships by solving an Ordinary Differential Equation and a Stochastic Differential Equation, respectively. Extensive experimentation on the Action Genome dataset validates the efficacy of the proposed methods.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

35 of 45 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2018Neural Ordinary Differential Equationscited by 3,733
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2022MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolationcited by 307
  6. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  7. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  8. 2021Anticipative Video Transformercited by 197
  9. 2018Stochastic Adversarial Video Predictioncited by 173
  10. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 163
  11. 2019Graph Neural Ordinary Differential Equationscited by 153
  12. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 146
23 more
  1. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  2. 2021SGTR: End-to-end Scene Graph Generation with Transformercited by 102
  3. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 95
  4. 2016An Uncertain Future: Forecasting from Static Images using Variational Autoencoderscited by 86
  5. 2020Semantic Image Manipulation Using Scene Graphscited by 82
  6. 20193-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agentscited by 74
  7. 2022Scene Graph Generation: A Comprehensive Surveycited by 73
  8. 2021Efficient and Accurate Gradients for Neural SDEscited by 68
  9. 2022Relationformer: A Unified Framework for Image-to-Graph Generationcited by 67
  10. 2020Learning Continuous System Dynamics from Irregularly-Sampled Partial Observationscited by 60
  11. 2022TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphscited by 59
  12. 2020Learning Long-term Visual Dynamics with Region Proposal Interaction Networkscited by 44
  13. 2020Vid-ODE: Continuous-Time Video Generation with Neural Ordinary Differential Equationcited by 39
  14. 2023Unbiased Scene Graph Generation in Videoscited by 38
  15. 2020Compositional Video Synthesis with Action Graphscited by 34
  16. 2021Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphscited by 30
  17. 2022(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answeringcited by 29
  18. 2023Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retentioncited by 29
  19. 2023Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Modelscited by 27
  20. 2023Modeling Dynamic Environments with Scene Graph Memorycited by 18
  21. 2020Modular Action Concept Grounding in Semantic Video Predictioncited by 12
  22. 2023VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generationcited by 12
  23. 2023Graph-based Multi-ODE Neural Networks for Spatio-Temporal Traffic Forecastingcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 2.

10 papers.

  1. 2024HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipationcited by 17
  2. 2024Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipationcited by 6
  3. 2026The Semantic Lifecycle in Embodied AI: Acquisition, Representation and Storage via Foundation Modelscited by 2
  4. 2025FORESCENE: FOREcasting human activity via latent SCENE graphs diffusioncited by 1
  5. 2025Position: Olfaction Standardization is Essential for the Advancement of Embodied Artificial Intelligencecited by 1
  6. 2025FDSG: Forecasting Dynamic Scene Graphscited by 1
  7. 2025Language-Driven Object-Oriented Two-Stage Method for Scene Graph Anticipationcited by 1
  8. 2025UNO: Unifying One-stage Video Scene Graph Generation via Object-Centric Visual Representation Learningcited by 1
  9. 2025Spatiotemporal Knowledge Graphs as Persistent Scene Memory for Embodied Question Answeringcited by 1
  10. 2026Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphscited by 0