mlsanityData to 14 Oct 2026

Spatial-Temporal Transformer for Dynamic Scene Graph Generation

arXiv 2107.12309 ↗First posted 26 Jul 2021cs.CVcited by 96

Not on the map. Why?

Abstract (arXiv)

Dynamic scene graph generation aims at generating a scene graph of the given video. Compared to the task of scene graph generation from images, it is more challenging because of the dynamic relationships between objects and the temporal dependencies between frames allowing for a richer semantic interpretation. In this paper, we propose Spatial-temporal Transformer (STTran), a neural network that consists of two core modules: (1) a spatial encoder that takes an input frame to extract spatial context and reason about the visual relationships within a frame, and (2) a temporal decoder which takes the output of the spatial encoder as input in order to capture the temporal dependencies between frames and infer the dynamic relationships. Furthermore, STTran is flexible to take varying lengths of videos as input without clipping, which is especially important for long videos. Our method is validated on the benchmark dataset Action Genome (AG). The experimental results demonstrate the superior performance of our method in terms of dynamic scene graphs. Moreover, a set of ablative studies is conducted and the effect of each proposed module is justified. Code available at: https://github.com/yrcong/STTran.

Key facts. Cited by 96 papers in the map (data through 14 Oct 2026). Builds on 51 of its 66 references that are in the map. First posted on arXiv 26 Jul 2021.

Builds on

51 of 66 references.

References · 51

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2017Mask R-CNNcited by 7,277
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  11. 2017Non-local Neural Networkscited by 2,293
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
38 more
  1. 2017The Kinetics Human Action Video Datasetcited by 1,909
  2. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  3. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  4. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  5. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  6. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  7. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  8. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 537
  9. 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 484
  10. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  11. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  12. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  13. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 414
  14. 2016Visual Relationship Detection with Language Priorscited by 334
  15. 2018Image Generation from Scene Graphscited by 324
  16. 2018Video Action Transformer Networkcited by 316
  17. 2018Graph R-CNN for Scene Graph Generationcited by 298
  18. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 259
  19. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 255
  20. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 250
  21. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  22. 2020GPS-Net: Graph Property Sensing Network for Scene Graph Generationcited by 166
  23. 2017Scene Graph Generation from Objects, Phrases and Region Captionscited by 154
  24. 2017Inferring and Executing Programs for Visual Reasoningcited by 130
  25. 2019Graphical Contrastive Losses for Scene Graph Parsingcited by 100
  26. 2017Detecting Visual Relationships with Deep Relational Networkscited by 97
  27. 2018Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generationcited by 78
  28. 2019Specifying Object Attributes and Relations in Interactive Scene Generationcited by 76
  29. 2017Visual Relationship Detection with Internal and External Linguistic Knowledge Distillationcited by 61
  30. 2020Lifted Disjoint Paths with Application in Multiple Object Trackingcited by 55
  31. 2021Context-Aware Layout to Image Generation with Enhanced Object Appearancecited by 41
  32. 2019Scene Graph Prediction with Limited Labelscited by 32
  33. 2020Image Captioning through Image Transformercited by 29
  34. 2017Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detectioncited by 28
  35. 2017Hidden Two-Stream Convolutional Networks for Action Recognitioncited by 24
  36. 2020NODIS: Neural Ordinary Differential Scene Understandingcited by 12
  37. 2016On Support Relations and Semantic Scene Graphscited by 9
  38. 2019Temporally Consistent Horizon Linescited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 16; 2023: 22; 2024: 21; 2025: 17; 2026: 17.

96 papers.

Sort:

By area:

  1. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 153
  2. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 96
  3. 2022Video Transformers: A Surveycited by 63
  4. 2022Delving into Sequential Patches for Deepfake Detectioncited by 41
  5. 2022Continuous Scene Representations for Embodied AIcited by 39
  6. 2023Unbiased Scene Graph Generation in Videoscited by 39
  7. 2022Latent Graph Representations for Critical View of Safety Assessmentcited by 37
  8. 2024AMEGO: Active Memory from long EGOcentric videoscited by 36
  9. 2022Compound Prototype Matching for Few-shot Action Recognitioncited by 35
  10. 2021Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphscited by 31
  11. 2022(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answeringcited by 30
  12. 2024STG-Mamba: Spatial-Temporal Graph Learning via Selective State Space Modelcited by 24
38 more
  1. 2021Classification-Then-Grounding: Reformulating Video Scene Graphs as Temporal Bipartite Graphscited by 23
  2. 2022STNDT: Modeling Neural Population Activity with a Spatiotemporal Transformercited by 23
  3. 2022Iterative Scene Graph Generation with Generative Transformerscited by 22
  4. 2023Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detectioncited by 22
  5. 20244D Panoptic Scene Graph Generationcited by 22
  6. 2022RNTrajRec: Road Network Enhanced Trajectory Recovery with Spatial-Temporal Transformercited by 21
  7. 2023LABRAD-OR: Lightweight Memory Scene Graphs for Accurate Bimodal Reasoning in Dynamic Operating Roomscited by 21
  8. 2022Meta Spatio-Temporal Debiasing for Video Scene Graph Generationcited by 20
  9. 2022GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinementcited by 20
  10. 2024HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipationcited by 19
  11. 2025Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMscited by 19
  12. 2021Rethinking the Two-Stage Framework for Grounded Situation Recognitioncited by 16
  13. 2023Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labelingcited by 15
  14. 2023Cross-Modality Time-Variant Relation Learning for Generating Dynamic Scene Graphscited by 14
  15. 2023LASER: A Neuro-Symbolic Framework for Learning Spatial-Temporal Scene Graphs with Weak Supervisioncited by 13
  16. 2023Human-Object Interaction Prediction in Videos through Gaze Followingcited by 13
  17. 2023HOI4ABOT: Human-Object Interaction Anticipation for Human Intention Reading Collaborative roBOTscited by 12
  18. 2022Scene Graph Expansion for Semantics-Guided Image Outpaintingcited by 11
  19. 2024CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videoscited by 11
  20. 2025DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generationcited by 11
  21. 2025Universal Scene Graph Generationcited by 11
  22. 2023Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generationcited by 10
  23. 2024Computer Vision for Primate Behavior Analysis in the Wildcited by 10
  24. 2024Towards Scene Graph Anticipationcited by 10
  25. 2024A Generalizable Anomaly Detection Method in Dynamic Graphscited by 10
  26. 2023Explainable Action Prediction through Self-Supervision on Scene Graphscited by 9
  27. 2022SSGVS: Semantic Scene Graph-to-Video Synthesiscited by 8
  28. 2023StageInteractor: Query-based Object Detector with Cross-stage Interactioncited by 8
  29. 2024Federated Transformer: Multi-Party Vertical Federated Learning on Practical Fuzzily Linked Datacited by 8
  30. 2023ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognitioncited by 6
  31. 2024Enhancing Human-Centered Dynamic Scene Understanding via Multiple LLMs Collaborated Reasoningcited by 6
  32. 2024Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipationcited by 6
  33. 2025Learning 4D Panoptic Scene Graph Generation from Rich 2D Visual Scenecited by 6
  34. 2024AUG: A New Dataset and An Efficient Model for Aerial Image Urban Scene Graph Generationcited by 5
  35. 2024CycleHOI: Improving Human-Object Interaction Detection with Cycle Consistency of Detection and Generationcited by 5
  36. 2024SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generationcited by 5
  37. 2026GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinkingcited by 5
  38. 2023Learning-based Relational Object Matching Across Viewscited by 4

50 most cited of 96 All 96 (JSON)