# Unbiased Scene Graph Generation via Two-stage Causal Modeling

arXiv: https://arxiv.org/abs/2307.05276 · HTML: https://mlsanity.com/paper/2307.05276/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 72 references that are in the map.
- First posted on arXiv 11 Jul 2023.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Decoupling Representation and Classifier for Long-Tailed Recognition](https://mlsanity.com/paper/1910.09217/) (2019)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Adapting Neural Networks for the Estimation of Treatment Effects](https://mlsanity.com/paper/1906.02120/) (2019)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms](https://mlsanity.com/paper/1901.10912/) (2019)
- [Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective](https://mlsanity.com/paper/2003.10780/) (2020)
- [Targeted Supervised Contrastive Learning for Long-Tailed Recognition](https://mlsanity.com/paper/2111.13998/) (2021)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Retrieval Augmented Classification for Long-Tail Visual Recognition](https://mlsanity.com/paper/2202.11233/) (2022)
- [Energy-Based Learning for Scene Graph Generation](https://mlsanity.com/paper/2103.02221/) (2021)
- [Explainable and Explicit Visual Reasoning over Scene Graphs](https://mlsanity.com/paper/1812.01855/) (2018)
- [Unpaired Image Captioning via Scene Graph Alignments](https://mlsanity.com/paper/1903.10658/) (2019)
- [Long-tailed Visual Recognition via Gaussian Clouded Logit Adjustment](https://mlsanity.com/paper/2305.11733/) (2023)
- [Recovering the Unbiased Scene Graphs from the Biased Ones](https://mlsanity.com/paper/2107.02112/) (2021)
- [A causal framework for distribution generalization](https://mlsanity.com/paper/2006.07433/) (2020)
- [Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2203.09811/) (2022)
- [Weakly Supervised Representation Learning with Sparse Perturbations](https://mlsanity.com/paper/2206.01101/) (2022)
- [CogTree: Cognition Tree Loss for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2009.07526/) (2020)

## Built on by (most cited first, 21 of 21)

- [LSKNet: A Foundation Lightweight Backbone for Remote Sensing](https://mlsanity.com/paper/2403.11735/) (2024)
- [STAR: A First-Ever Dataset and A Large-Scale Benchmark for Scene Graph Generation in Large-Size Satellite Imagery](https://mlsanity.com/paper/2406.09410/) (2024)
- [SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection](https://mlsanity.com/paper/2412.20665/) (2024)
- [Robust Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoning](https://mlsanity.com/paper/2502.12425/) (2025)
- [HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation](https://mlsanity.com/paper/2403.12033/) (2024)
- [A Causal Adjustment Module for Debiasing Scene Graph Generation](https://mlsanity.com/paper/2503.17862/) (2025)
- [Leveraging Predicate and Triplet Learning for Scene Graph Generation](https://mlsanity.com/paper/2406.02038/) (2024)
- [From Laboratory to Real World: A New Benchmark Towards Privacy-Preserved Visible-Infrared Person Re-Identification](https://mlsanity.com/paper/2503.12232/) (2025)
- [Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models](https://mlsanity.com/paper/2505.19474/) (2025)
- [Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution Can Improve Robust Scene Graph Generation](https://mlsanity.com/paper/2504.12606/) (2025)
- [MST: Adaptive Multi-Scale Tokens Guided Interactive Segmentation](https://mlsanity.com/paper/2401.04403/) (2024)
- [Empowering Vision Transformers with Multi-Scale Causal Intervention for Long-Tailed Image Classification](https://mlsanity.com/paper/2505.08173/) (2025)
- [From Data to Modeling: Fully Open-vocabulary Scene Graph Generation](https://mlsanity.com/paper/2505.20106/) (2025)
- [SGTR+: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2401.12835/) (2024)
- [A Reverse Causal Framework to Mitigate Spurious Correlations for Debiasing Scene Graph Generation](https://mlsanity.com/paper/2505.23451/) (2025)
- [Dual-domain Adaptation Networks for Realistic Image Super-resolution](https://mlsanity.com/paper/2511.17217/) (2025)
- [ALF: Adaptive Label Finetuning for Scene Graph Generation](https://mlsanity.com/paper/2312.17425/) (2023)
- [Towards Causal Relationship in Indefinite Data: Baseline Model and New Datasets](https://mlsanity.com/paper/2401.08221/) (2024)
- [A Closer Look at Dynamic Scene Graph Generation In the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2503.15846/) (2025)
- [BUSSARD: Normalizing Flows for Bijective Universal Scene-Specific Anomalous Relationship Detection](https://mlsanity.com/paper/2603.16645/) (2026)
- [T-STAR: A Large-Scale Benchmark for Spatio-Temporal Panoptic Scene Graph Generation in Satellite Video](https://mlsanity.com/paper/2607.21228/) (2026)

## Abstract (arXiv)

Despite the impressive performance of recent unbiased Scene Graph Generation (SGG) methods, the current debiasing literature mainly focuses on the long-tailed distribution problem, whereas it overlooks another source of bias, i.e., semantic confusion, which makes the SGG model prone to yield false predictions for similar relationships. In this paper, we explore a debiasing procedure for the SGG task leveraging causal inference. Our central insight is that the Sparse Mechanism Shift (SMS) in causality allows independent intervention on multiple biases, thereby potentially preserving head category performance while pursuing the prediction of high-informative tail relationships. However, the noisy datasets lead to unobserved confounders for the SGG task, and thus the constructed causal models are always causal-insufficient to benefit from SMS. To remedy this, we propose Two-stage Causal Modeling (TsCM) for the SGG task, which takes the long-tailed distribution and semantic confusion as confounders to the Structural Causal Model (SCM) and then decouples the causal intervention into two stages. The first stage is causal representation learning, where we use a novel Population Loss (P-Loss) to intervene in the semantic confusion confounder. The second stage introduces the Adaptive Logit Adjustment (AL-Adjustment) to eliminate the long-tailed distribution confounder to complete causal calibration learning. These two stages are model agnostic and thus can be used in any SGG model that seeks unbiased predictions. Comprehensive experiments conducted on the popular SGG backbones and benchmarks show that our TsCM can achieve state-of-the-art performance in terms of mean recall rate. Furthermore, TsCM can maintain a higher recall rate than other debiasing methods, which indicates that our method can achieve a better tradeoff between head and tail relationships.