# From General to Specific: Informative Scene Graph Generation via Balance Adjustment

arXiv: https://arxiv.org/abs/2108.13129 · HTML: https://mlsanity.com/paper/2108.13129/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 41 references that are in the map.
- First posted on arXiv 30 Aug 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](https://mlsanity.com/paper/2003.12962/) (2020)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [Visual Translation Embedding Network for Visual Relation Detection](https://mlsanity.com/paper/1702.08319/) (2017)
- [Scene Graph Generation with External Knowledge and Image Reconstruction](https://mlsanity.com/paper/1904.00560/) (2019)
- [Bridging Knowledge Graphs to Generate Scene Graphs](https://mlsanity.com/paper/2001.02314/) (2020)
- [Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation](https://mlsanity.com/paper/1806.11538/) (2018)
- [Boosting Image Captioning with Attributes](https://mlsanity.com/paper/1611.01646/) (2016)
- [Pixels to Graphs by Associative Embedding](https://mlsanity.com/paper/1706.07365/) (2017)
- [Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction](https://mlsanity.com/paper/1802.05451/) (2018)
- [Pragmatically Informative Image Captioning with Character-Level Inference](https://mlsanity.com/paper/1804.05417/) (2018)
- [Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition](https://mlsanity.com/paper/1807.04979/) (2018)
- [Relation Transformer Network](https://mlsanity.com/paper/2004.06193/) (2020)
- [Informative Image Captioning with External Sources of Information](https://mlsanity.com/paper/1906.08876/) (2019)

## Built on by (most cited first, 30 of 51)

- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [SGTR: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2112.12970/) (2021)
- [RelTR: Relation Transformer for Scene Graph Generation](https://mlsanity.com/paper/2201.11460/) (2022)
- [The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generation](https://mlsanity.com/paper/2206.03014/) (2022)
- [Fine-Grained Scene Graph Generation with Data Transfer](https://mlsanity.com/paper/2203.11654/) (2022)
- [From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models](https://mlsanity.com/paper/2404.00906/) (2024)
- [Prototype-based Embedding Network for Scene Graph Generation](https://mlsanity.com/paper/2303.07096/) (2023)
- [SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text](https://mlsanity.com/paper/2204.11964/) (2022)
- [RU-Net: Regularized Unrolling Network for Scene Graph Generation](https://mlsanity.com/paper/2205.01297/) (2022)
- [Fine-Grained Predicates Learning for Scene Graph Generation](https://mlsanity.com/paper/2204.02597/) (2022)
- [Iterative Scene Graph Generation](https://mlsanity.com/paper/2207.13440/) (2022)
- [Devil's on the Edges: Selective Quad Attention for Scene Graph Generation](https://mlsanity.com/paper/2304.03495/) (2023)
- [Iterative Scene Graph Generation with Generative Transformers](https://mlsanity.com/paper/2211.16636/) (2022)
- [Unbiased Scene Graph Generation via Two-stage Causal Modeling](https://mlsanity.com/paper/2307.05276/) (2023)
- [Meta Spatio-Temporal Debiasing for Video Scene Graph Generation](https://mlsanity.com/paper/2207.11441/) (2022)
- [NICEST: Noisy Label Correction and Training for Robust Scene Graph Generation](https://mlsanity.com/paper/2207.13316/) (2022)
- [HiLo: Exploiting High Low Frequency Relations for Unbiased Panoptic Scene Graph Generation](https://mlsanity.com/paper/2303.15994/) (2023)
- [Pair then Relation: Pair-Net for Panoptic Scene Graph Generation](https://mlsanity.com/paper/2307.08699/) (2023)
- [Hierarchical Memory Learning for Fine-Grained Scene Graph Generation](https://mlsanity.com/paper/2203.06907/) (2022)
- [Compositional Feature Augmentation for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2308.06712/) (2023)
- [Adaptive Fine-Grained Predicates Learning for Scene Graph Generation](https://mlsanity.com/paper/2207.04602/) (2022)
- [Label Semantic Knowledge Distillation for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2208.03763/) (2022)
- [HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation](https://mlsanity.com/paper/2403.12033/) (2024)
- [A Causal Adjustment Module for Debiasing Scene Graph Generation](https://mlsanity.com/paper/2503.17862/) (2025)
- [Adaptive Self-training Framework for Fine-grained Scene Graph Generation](https://mlsanity.com/paper/2401.09786/) (2024)
- [Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection](https://mlsanity.com/paper/2403.17709/) (2024)
- [Dual-branch Hybrid Learning Network for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2207.07913/) (2022)
- [Environment-Invariant Curriculum Relation Learning for Fine-Grained Scene Graph Generation](https://mlsanity.com/paper/2308.03282/) (2023)
- [Probabilistic Debiasing of Scene Graphs](https://mlsanity.com/paper/2211.06444/) (2022)
- [Leveraging Predicate and Triplet Learning for Scene Graph Generation](https://mlsanity.com/paper/2406.02038/) (2024)

## Abstract (arXiv)

The scene graph generation (SGG) task aims to detect visual relationship triplets, i.e., subject, predicate, object, in an image, providing a structural vision layout for scene understanding. However, current models are stuck in common predicates, e.g., "on" and "at", rather than informative ones, e.g., "standing on" and "looking at", resulting in the loss of precise information and overall performance. If a model only uses "stone on road" rather than "blocking" to describe an image, it is easy to misunderstand the scene. We argue that this phenomenon is caused by two key imbalances between informative predicates and common ones, i.e., semantic space level imbalance and training sample level imbalance. To tackle this problem, we propose BA-SGG, a simple yet effective SGG framework based on balance adjustment but not the conventional distribution fitting. It integrates two components: Semantic Adjustment (SA) and Balanced Predicate Learning (BPL), respectively for adjusting these imbalances. Benefited from the model-agnostic process, our method is easily applied to the state-of-the-art SGG models and significantly improves the SGG performance. Our method achieves 14.3%, 8.0%, and 6.1% higher Mean Recall (mR) than that of the Transformer model at three scene graph generation sub-tasks on Visual Genome, respectively. Codes are publicly available.