# Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphs

arXiv: https://arxiv.org/abs/2112.09828 · HTML: https://mlsanity.com/paper/2112.09828/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 18 Dec 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [Exploring Visual Relationship for Image Captioning](https://mlsanity.com/paper/1809.07041/) (2018)
- [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](https://mlsanity.com/paper/2003.12962/) (2020)
- [Scene Graph Generation from Objects, Phrases and Region Captions](https://mlsanity.com/paper/1707.09700/) (2017)
- [Graphical Contrastive Losses for Scene Graph Parsing](https://mlsanity.com/paper/1903.02728/) (2019)
- [Spatial-Temporal Transformer for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2107.12309/) (2021)
- [Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval](https://mlsanity.com/paper/1910.05134/) (2019)
- [Target Adaptive Context Aggregation for Video Scene Graph Generation](https://mlsanity.com/paper/2108.08121/) (2021)
- [Learnable Graph Matching: Incorporating Graph Partitioning with Deep Feature Learning for Multiple Object Tracking](https://mlsanity.com/paper/2103.16178/) (2021)
- [PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph](https://mlsanity.com/paper/1905.01608/) (2019)
- [Unified Graph Structured Models for Video Understanding](https://mlsanity.com/paper/2103.15662/) (2021)
- [RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval](https://mlsanity.com/paper/2007.08513/) (2020)
- [In Defense of Scene Graphs for Image Captioning](https://mlsanity.com/paper/2102.04990/) (2021)

## Built on by (most cited first, 30 of 31)

- [Action Scene Graphs for Long-Form Understanding of Egocentric Videos](https://mlsanity.com/paper/2312.03391/) (2023)
- [OED: Towards One-stage End-to-End Dynamic Scene Graph Generation](https://mlsanity.com/paper/2405.16925/) (2024)
- [HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation](https://mlsanity.com/paper/2411.18042/) (2024)
- [Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labeling](https://mlsanity.com/paper/2308.05081/) (2023)
- [Cross-Modality Time-Variant Relation Learning for Generating Dynamic Scene Graphs](https://mlsanity.com/paper/2305.08522/) (2023)
- [DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generation](https://mlsanity.com/paper/2503.13957/) (2025)
- [Universal Scene Graph Generation](https://mlsanity.com/paper/2503.15005/) (2025)
- [Computer Vision for Primate Behavior Analysis in the Wild](https://mlsanity.com/paper/2401.16424/) (2024)
- [Towards Scene Graph Anticipation](https://mlsanity.com/paper/2403.04899/) (2024)
- [A Hierarchical Graph-based Approach for Recognition and Description Generation of Bimanual Actions in Videos](https://mlsanity.com/paper/2310.00670/) (2023)
- [Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation](https://mlsanity.com/paper/2411.13059/) (2024)
- [Situational Scene Graph for Structured Human-centric Situation Understanding](https://mlsanity.com/paper/2410.22829/) (2024)
- [Weakly Supervised Video Scene Graph Generation via Natural Language Supervision](https://mlsanity.com/paper/2502.15370/) (2025)
- [DDS: Decoupled Dynamic Scene-Graph Generation Network](https://mlsanity.com/paper/2301.07666/) (2023)
- [Taking A Closer Look at Visual Relation: Unbiased Video Scene Graph Generation with Decoupled Label Learning](https://mlsanity.com/paper/2303.13209/) (2023)
- [TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferring](https://mlsanity.com/paper/2508.04943/) (2025)
- [UNO: Unifying One-stage Video Scene Graph Generation via Object-Centric Visual Representation Learning](https://mlsanity.com/paper/2509.06165/) (2025)
- [Triple Correlations-Guided Label Supplementation for Unbiased Video Scene Graph Generation](https://mlsanity.com/paper/2307.16309/) (2023)
- [TD^2-Net: Toward Denoising and Debiasing for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2401.12479/) (2024)
- [FORESCENE: FOREcasting human activity via latent SCENE graphs diffusion](https://mlsanity.com/paper/2503.06182/) (2025)
- [FDSG: Forecasting Dynamic Scene Graphs](https://mlsanity.com/paper/2506.01487/) (2025)
- [Language-Driven Object-Oriented Two-Stage Method for Scene Graph Anticipation](https://mlsanity.com/paper/2509.05661/) (2025)
- [VOST-SGG: VLM-Aided One-Stage Spatio-Temporal Scene Graph Generation](https://mlsanity.com/paper/2512.05524/) (2025)
- [Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learning](https://mlsanity.com/paper/2603.21559/) (2026)
- [SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models](https://mlsanity.com/paper/2605.13667/) (2026)
- [A Closer Look at Dynamic Scene Graph Generation In the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2503.15846/) (2025)
- [THYME: Temporal Hierarchical-Cyclic Interactivity Modeling for Video Scene Graphs in Aerial Footage](https://mlsanity.com/paper/2507.09200/) (2025)
- [Click2Graph: Interactive Panoptic Video Scene Graphs from a Single Click](https://mlsanity.com/paper/2511.15948/) (2025)
- [Frequency-guided Multi-level Reasoning for Scene Graph Generation in Video](https://mlsanity.com/paper/2604.17298/) (2026)
- [OvDSGG: End-to-End Open-Vocabulary Dynamic Scene Graph Generation](https://mlsanity.com/paper/2608.14835/) (2026)

## Abstract (arXiv)

Dynamic scene graph generation from a video is challenging due to the temporal dynamics of the scene and the inherent temporal fluctuations of predictions. We hypothesize that capturing long-term temporal dependencies is the key to effective generation of dynamic scene graphs. We propose to learn the long-term dependencies in a video by capturing the object-level consistency and inter-object relationship dynamics over object-level long-term tracklets using transformers. Experimental results demonstrate that our Dynamic Scene Graph Detection Transformer (DSG-DETR) outperforms state-of-the-art methods by a significant margin on the benchmark dataset Action Genome. Our ablation studies validate the effectiveness of each component of the proposed approach. The source code is available at https://github.com/Shengyu-Feng/DSG-DETR.