# TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferring

arXiv: https://arxiv.org/abs/2508.04943 · HTML: https://mlsanity.com/paper/2508.04943/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 36 references that are in the map.
- First posted on arXiv 7 Aug 2025.

## Builds on (most cited first, 22 of 22)

- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [TIDE: A General Toolbox for Identifying Object Detection Errors](https://mlsanity.com/paper/2008.08115/) (2020)
- [RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection](https://mlsanity.com/paper/2209.01814/) (2022)
- [Target Adaptive Context Aggregation for Video Scene Graph Generation](https://mlsanity.com/paper/2108.08121/) (2021)
- [RLIPv2: Fast Scaling of Relational Language-Image Pre-training](https://mlsanity.com/paper/2308.09351/) (2023)
- [Towards Precise End-to-end Weakly Supervised Object Detection Network](https://mlsanity.com/paper/1911.12148/) (2019)
- [Unified Graph Structured Models for Video Understanding](https://mlsanity.com/paper/2103.15662/) (2021)
- [Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memory](https://mlsanity.com/paper/2309.03696/) (2023)
- [Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphs](https://mlsanity.com/paper/2112.09828/) (2021)
- [Training-free Video Temporal Grounding using Large-scale Pre-trained Models](https://mlsanity.com/paper/2408.16219/) (2024)
- [Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detection](https://mlsanity.com/paper/2408.02484/) (2024)
- [Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection](https://mlsanity.com/paper/2404.06194/) (2024)
- [Classification-Then-Grounding: Reformulating Video Scene Graphs as Temporal Bipartite Graphs](https://mlsanity.com/paper/2112.04222/) (2021)
- [Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detection](https://mlsanity.com/paper/2302.00268/) (2023)
- [Bridging the Gap between 2D and 3D Visual Question Answering: A Fusion Approach for 3D VQA](https://mlsanity.com/paper/2402.15933/) (2024)
- [3D Vision and Language Pretraining with Large-Scale Synthetic Data](https://mlsanity.com/paper/2407.06084/) (2024)
- [Video Relation Detection via Tracklet based Visual Transformer](https://mlsanity.com/paper/2108.08669/) (2021)
- [Activity Driven Weakly Supervised Object Detection](https://mlsanity.com/paper/1904.01665/) (2019)
- [Active Object Detection with Knowledge Aggregation and Distillation from Large Models](https://mlsanity.com/paper/2405.12509/) (2024)
- [ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding](https://mlsanity.com/paper/2408.16314/) (2024)
- [Weakly Supervised Video Scene Graph Generation via Natural Language Supervision](https://mlsanity.com/paper/2502.15370/) (2025)

## Built on by (most cited first, 3 of 3)

- [Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learning](https://mlsanity.com/paper/2603.21559/) (2026)
- [Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild](https://mlsanity.com/paper/2607.13881/) (2026)
- [Rule-Compliant Visual Spatial Planning for Multimodal Large Language Models](https://mlsanity.com/paper/2608.20237/) (2026)

## Abstract (arXiv)

Dynamic Scene Graph Generation (DSGG) aims to create a scene graph for each video frame by detecting objects and predicting their relationships. Weakly Supervised DSGG (WS-DSGG) reduces annotation workload by using an unlocalized scene graph from a single frame per video for training. Existing WS-DSGG methods depend on an off-the-shelf external object detector to generate pseudo labels for subsequent DSGG training. However, detectors trained on static, object-centric images struggle in dynamic, relation-aware scenarios required for DSGG, leading to inaccurate localization and low-confidence proposals. To address the challenges posed by external object detectors in WS-DSGG, we propose a Temporal-enhanced Relation-aware Knowledge Transferring (TRKT) method, which leverages knowledge to enhance detection in relation-aware dynamic scenarios. TRKT is built on two key components:(1)Relation-aware knowledge mining: we first employ object and relation class decoders that generate category-specific attention maps to highlight both object regions and interactive areas. Then we propose an Inter-frame Attention Augmentation strategy that exploits optical flow for neighboring frames to enhance the attention maps, making them motion-aware and robust to motion blur. This step yields relation- and motion-aware knowledge mining for WS-DSGG. (2) we introduce a Dual-stream Fusion Module that integrates category-specific attention maps into external detections to refine object localization and boost confidence scores for object proposals. Extensive experiments demonstrate that TRKT achieves state-of-the-art performance on Action Genome dataset. Our code is avaliable at https://github.com/XZPKU/TRKT.git.