# Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation

arXiv: https://arxiv.org/abs/2411.13059 · HTML: https://mlsanity.com/paper/2411.13059/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 65 references that are in the map.
- First posted on arXiv 20 Nov 2024.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [On Evaluating Adversarial Robustness](https://mlsanity.com/paper/1902.06705/) (2019)
- [BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/1912.02413/) (2019)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [Equalization Loss for Long-Tailed Object Recognition](https://mlsanity.com/paper/2003.05176/) (2020)
- [Long-tailed Recognition by Routing Diverse Distribution-Aware Experts](https://mlsanity.com/paper/2010.01809/) (2020)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [Teacher-Student Curriculum Learning](https://mlsanity.com/paper/1707.00183/) (2017)
- [Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification](https://mlsanity.com/paper/2001.01536/) (2020)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets](https://mlsanity.com/paper/2007.09654/) (2020)
- [Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks](https://mlsanity.com/paper/1802.03796/) (2018)
- [Range Loss for Deep Face Recognition with Long-tail](https://mlsanity.com/paper/1611.08976/) (2016)
- [Spatial-Temporal Transformer for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2107.12309/) (2021)
- [Long-tailed Visual Recognition via Gaussian Clouded Logit Adjustment](https://mlsanity.com/paper/2305.11733/) (2023)
- [3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents](https://mlsanity.com/paper/1908.04929/) (2019)
- [Factors in Finetuning Deep Model for object detection](https://mlsanity.com/paper/1601.05150/) (2016)
- [How hard can it be? Estimating the difficulty of visual search in an image](https://mlsanity.com/paper/1705.08280/) (2017)

## Built on by (most cited first, 6 of 6)

- [From Pixels to Graphs: using Scene and Knowledge Graphs for HD-EPIC VQA Challenge](https://mlsanity.com/paper/2506.08553/) (2025)
- [IMPACT-CYCLE: A Contract-Based Multi-Agent System for Claim-Level Supervisory Correction of Long-Video Semantic Memory](https://mlsanity.com/paper/2604.20136/) (2026)
- [Position: Olfaction Standardization is Essential for the Advancement of Embodied Artificial Intelligence](https://mlsanity.com/paper/2506.00398/) (2025)
- [Language-Driven Object-Oriented Two-Stage Method for Scene Graph Anticipation](https://mlsanity.com/paper/2509.05661/) (2025)
- [Graph it first! Enabling Reasoning on Long-form Egocentric Videos through Scene Graphs](https://mlsanity.com/paper/2606.25842/) (2026)
- [SenBen: Sensitive Scene Graphs for Explainable Content Moderation](https://mlsanity.com/paper/2604.08819/) (2026)

## Abstract (arXiv)

Spatio-Temporal Scene Graphs (STSGs) provide a concise and expressive representation of dynamic scenes by modeling objects and their evolving relationships over time. However, real-world visual relationships often exhibit a long-tailed distribution, causing existing methods for tasks like Video Scene Graph Generation (VidSGG) and Scene Graph Anticipation (SGA) to produce biased scene graphs. To this end, we propose ImparTail, a novel training framework that leverages loss masking and curriculum learning to mitigate bias in the generation and anticipation of spatio-temporal scene graphs. Unlike prior methods that add extra architectural components to learn unbiased estimators, we propose an impartial training objective that reduces the dominance of head classes during learning and focuses on underrepresented tail relationships. Our curriculum-driven mask generation strategy further empowers the model to adaptively adjust its bias mitigation strategy over time, enabling more balanced and robust estimations. To thoroughly assess performance under various distribution shifts, we also introduce two new tasks Robust Spatio-Temporal Scene Graph Generation and Robust Scene Graph Anticipation offering a challenging benchmark for evaluating the resilience of STSG models. Extensive experiments on the Action Genome dataset demonstrate the superior unbiased performance and robustness of our method compared to existing baselines.