# Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation

arXiv: https://arxiv.org/abs/2412.07160 · HTML: https://mlsanity.com/paper/2412.07160/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Builds on (most cited first, 30 of 31)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions](https://mlsanity.com/paper/2004.03967/) (2020)
- [Transformer-Based Visual Segmentation: A Survey](https://mlsanity.com/paper/2304.09854/) (2023)
- [Video K-Net: A Simple, Strong, and Unified Baseline for Video Segmentation](https://mlsanity.com/paper/2204.04656/) (2022)
- [Do Different Tracking Tasks Require Different Appearance Models?](https://mlsanity.com/paper/2107.02156/) (2021)
- [Efficient Video Prediction via Sparsely Conditioned Flow Matching](https://mlsanity.com/paper/2211.14575/) (2022)
- [MemFlow: Optical Flow Estimation and Prediction with Memory](https://mlsanity.com/paper/2404.04808/) (2024)
- [Action Scene Graphs for Long-Form Understanding of Egocentric Videos](https://mlsanity.com/paper/2312.03391/) (2023)
- [Contrastive Learning for Neural Topic Model](https://mlsanity.com/paper/2110.12764/) (2021)
- [Unbiased Scene Graph Generation in Videos](https://mlsanity.com/paper/2304.00733/) (2023)
- [Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives](https://mlsanity.com/paper/2406.05615/) (2024)
- [InfoCTM: A Mutual Information Maximization Perspective of Cross-Lingual Topic Modeling](https://mlsanity.com/paper/2304.03544/) (2023)
- [Tube-Link: A Flexible Cross Tube Framework for Universal Video Segmentation](https://mlsanity.com/paper/2303.12782/) (2023)
- [Modeling Dynamic Topics in Chain-Free Fashion by Evolution-Tracking Contrastive Learning and Unassociated Word Exclusion](https://mlsanity.com/paper/2405.17957/) (2024)
- [TextPSG: Panoptic Scene Graph Generation from Textual Descriptions](https://mlsanity.com/paper/2310.07056/) (2023)
- [KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning](https://mlsanity.com/paper/2403.17486/) (2024)
- [Improving Multimodal Sentiment Analysis: Supervised Angular Margin-based Contrastive Learning for Enhanced Fusion Representation](https://mlsanity.com/paper/2312.02227/) (2023)
- [Topic Modeling as Multi-Objective Contrastive Optimization](https://mlsanity.com/paper/2402.07577/) (2024)
- [DemaFormer: Damped Exponential Moving Average Transformer with Energy-Based Modeling for Temporal Language Grounding](https://mlsanity.com/paper/2312.02549/) (2023)
- [Encoding and Controlling Global Semantics for Long-form Video Question Answering](https://mlsanity.com/paper/2405.19723/) (2024)
- [MAMA: Meta-optimized Angular Margin Contrastive Framework for Video-Language Representation Learning](https://mlsanity.com/paper/2407.03788/) (2024)
- [ARVideo: Autoregressive Pretraining for Self-Supervised Video Representation Learning](https://mlsanity.com/paper/2405.15160/) (2024)
- [Taking A Closer Look at Visual Relation: Unbiased Video Scene Graph Generation with Decoupled Label Learning](https://mlsanity.com/paper/2303.13209/) (2023)
- [Video Enriched Retrieval Augmented Generation Using Aligned Video Captions](https://mlsanity.com/paper/2405.17706/) (2024)
- [An Animation-based Augmentation Approach for Action Recognition from Discontinuous Video](https://mlsanity.com/paper/2404.06741/) (2024)

## Built on by (most cited first, 9 of 9)

- [TI-JEPA: An Innovative Energy-based Joint Embedding Strategy for Text-Image Multimodal Systems](https://mlsanity.com/paper/2503.06380/) (2025)
- [SAMJAM: Zero-Shot Video Scene Graph Generation for Egocentric Kitchen Videos](https://mlsanity.com/paper/2504.07867/) (2025)
- [UNO: Unifying One-stage Video Scene Graph Generation via Object-Centric Visual Representation Learning](https://mlsanity.com/paper/2509.06165/) (2025)
- [Temporal-Oriented Recipe for Transferring Large Vision-Language Model to Video Understanding](https://mlsanity.com/paper/2505.12605/) (2025)
- [M3DMap: Object-aware Multimodal 3D Mapping for Dynamic Environments](https://mlsanity.com/paper/2508.17044/) (2025)
- [More Bias, Less Bias: BiasPrompting for Enhanced Multiple-Choice Question Answering](https://mlsanity.com/paper/2511.20086/) (2025)
- [A Closer Look at Dynamic Scene Graph Generation In the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2503.15846/) (2025)
- [Click2Graph: Interactive Panoptic Video Scene Graphs from a Single Click](https://mlsanity.com/paper/2511.15948/) (2025)
- [T-STAR: A Large-Scale Benchmark for Spatio-Temporal Panoptic Scene Graph Generation in Satellite Video](https://mlsanity.com/paper/2607.21228/) (2026)

## Abstract (arXiv)

To equip artificial intelligence with a comprehensive understanding towards a temporal world, video and 4D panoptic scene graph generation abstracts visual data into nodes to represent entities and edges to capture temporal relations. Existing methods encode entity masks tracked across temporal dimensions (mask tubes), then predict their relations with temporal pooling operation, which does not fully utilize the motion indicative of the entities' relation. To overcome this limitation, we introduce a contrastive representation learning framework that focuses on motion pattern for temporal scene graph generation. Firstly, our framework encourages the model to learn close representations for mask tubes of similar subject-relation-object triplets. Secondly, we seek to push apart mask tubes from their temporally shuffled versions. Moreover, we also learn distant representations for mask tubes belonging to the same video but different triplets. Extensive experiments show that our motion-aware contrastive framework significantly improves state-of-the-art methods on both video and 4D datasets. Code is available at: https://github.com/nguyentthong/motion-contrastive-sgg