# Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation

arXiv: https://arxiv.org/abs/2306.10122 · HTML: https://mlsanity.com/paper/2306.10122/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 47 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Deep Long-Tailed Learning: A Survey](https://mlsanity.com/paper/2110.04596/) (2021)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [YFCC100M: The New Data in Multimedia Research](https://mlsanity.com/paper/1503.01817/) (2015)
- [Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2104.00308/) (2021)
- [Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets](https://mlsanity.com/paper/2007.09654/) (2020)
- [Spatial-Temporal Transformer for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2107.12309/) (2021)
- [PCPL: Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2009.00893/) (2020)
- [Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2203.09811/) (2022)
- [Learning of Visual Relations: The Devil is in the Tails](https://mlsanity.com/paper/2108.09668/) (2021)
- [Fully Convolutional Scene Graph Generation](https://mlsanity.com/paper/2103.16083/) (2021)
- [Classification-Then-Grounding: Reformulating Video Scene Graphs as Temporal Bipartite Graphs](https://mlsanity.com/paper/2112.04222/) (2021)
- [Learning Interactions and Relationships between Movie Characters](https://mlsanity.com/paper/2003.13158/) (2020)
- [Striking the Right Balance: Recall Loss for Semantic Segmentation](https://mlsanity.com/paper/2106.14917/) (2021)
- [Social Fabric: Tubelet Compositions for Video Relation Detection](https://mlsanity.com/paper/2108.08363/) (2021)
- [LIGHTEN: Learning Interactions with Graph and Hierarchical TEmporal Networks for HOI in videos](https://mlsanity.com/paper/2012.09402/) (2020)
- [Predicate correlation learning for scene graph generation](https://mlsanity.com/paper/2107.02713/) (2021)
- [EvoGrad: Efficient Gradient-Based Meta-Learning and Hyperparameter Optimization](https://mlsanity.com/paper/2106.10575/) (2021)
- [Video Relation Detection with Trajectory-aware Multi-modal Features](https://mlsanity.com/paper/2101.08165/) (2021)
- [Diagnosing Errors in Video Relation Detectors](https://mlsanity.com/paper/2110.13110/) (2021)

## Built on by (most cited first, 3 of 3)

- [A Causal Adjustment Module for Debiasing Scene Graph Generation](https://mlsanity.com/paper/2503.17862/) (2025)
- [A Reverse Causal Framework to Mitigate Spurious Correlations for Debiasing Scene Graph Generation](https://mlsanity.com/paper/2505.23451/) (2025)
- [CAGE-SGG: Counterfactual Active Graph Evidence for Open-Vocabulary Scene Graph Generation](https://mlsanity.com/paper/2604.22274/) (2026)

## Abstract (arXiv)

This paper investigates the problem of scene graph generation in videos with the aim of capturing semantic relations between subjects and objects in the form of langlesubject, predicate, objectrangle triplets. Recognizing the predicate between subject and object pairs is imbalanced and multi-label in nature, ranging from ubiquitous interactions such as spatial relationships (eg in front of) to rare interactions such as twisting. In widely-used benchmarks such as Action Genome and VidOR, the imbalance ratio between the most and least frequent predicates reaches 3,218 and 3,408, respectively, surpassing even benchmarks specifically designed for long-tailed recognition. Due to the long-tailed distributions and label co-occurrences, recent state-of-the-art methods predominantly focus on the most frequently occurring predicate classes, ignoring those in the long tail. In this paper, we analyze the limitations of current approaches for scene graph generation in videos and identify a one-to-one correspondence between predicate frequency and recall performance. To make the step towards unbiased scene graph generation in videos, we introduce a multi-label meta-learning framework to deal with the biased predicate distribution. Our meta-learning framework learns a meta-weight network for each training sample over all possible label losses. We evaluate our approach on the Action Genome and VidOR benchmarks by building upon two current state-of-the-art methods for each benchmark. The experiments demonstrate that the multi-label meta-weight network improves the performance for predicates in the long tail without compromising performance for head classes, resulting in better overall performance and favorable generalizability. Code: urlhttps://github.com/shanshuo/ML-MWN.