# Fine-Grained Scene Graph Generation with Data Transfer

arXiv: https://arxiv.org/abs/2203.11654 · HTML: https://mlsanity.com/paper/2203.11654/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 42 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 30 of 34)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)
- [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](https://mlsanity.com/paper/2003.12962/) (2020)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [Scene Graph Generation from Objects, Phrases and Region Captions](https://mlsanity.com/paper/1707.09700/) (2017)
- [Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2104.00308/) (2021)
- [Graph-Structured Representations for Visual Question Answering](https://mlsanity.com/paper/1609.05600/) (2016)
- [Energy-Based Learning for Scene Graph Generation](https://mlsanity.com/paper/2103.02221/) (2021)
- [Bridging Knowledge Graphs to Generate Scene Graphs](https://mlsanity.com/paper/2001.02314/) (2020)
- [Unpaired Image Captioning via Scene Graph Alignments](https://mlsanity.com/paper/1903.10658/) (2019)
- [PCPL: Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2009.00893/) (2020)
- [Recovering the Unbiased Scene Graphs from the Biased Ones](https://mlsanity.com/paper/2107.02112/) (2021)
- [Video as Conditional Graph Hierarchy for Multi-Granular Question Answering](https://mlsanity.com/paper/2112.06197/) (2021)
- [Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval](https://mlsanity.com/paper/1910.05134/) (2019)
- [Learning of Visual Relations: The Devil is in the Tails](https://mlsanity.com/paper/2108.09668/) (2021)
- [CogTree: Cognition Tree Loss for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2009.07526/) (2020)
- [From General to Specific: Informative Scene Graph Generation via Balance Adjustment](https://mlsanity.com/paper/2108.13129/) (2021)
- [NBDT: Neural-Backed Decision Trees](https://mlsanity.com/paper/2004.00221/) (2020)
- [Large-Scale Visual Relationship Understanding](https://mlsanity.com/paper/1804.10660/) (2018)

## Built on by (most cited first, 30 of 57)

- [Exploring Diverse In-Context Configurations for Image Captioning](https://mlsanity.com/paper/2305.14800/) (2023)
- [From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models](https://mlsanity.com/paper/2404.00906/) (2024)
- [Panoptic Scene Graph Generation with Semantics-Prototype Learning](https://mlsanity.com/paper/2307.15567/) (2023)
- [Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World](https://mlsanity.com/paper/2303.13233/) (2023)
- [How to Configure Good In-Context Sequence for Visual Question Answering](https://mlsanity.com/paper/2312.01571/) (2023)
- [PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models](https://mlsanity.com/paper/2205.11169/) (2022)
- [LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation](https://mlsanity.com/paper/2310.10404/) (2023)
- [Devil's on the Edges: Selective Quad Attention for Scene Graph Generation](https://mlsanity.com/paper/2304.03495/) (2023)
- [Unbiased Scene Graph Generation via Two-stage Causal Modeling](https://mlsanity.com/paper/2307.05276/) (2023)
- [HiLo: Exploiting High Low Frequency Relations for Unbiased Panoptic Scene Graph Generation](https://mlsanity.com/paper/2303.15994/) (2023)
- [Vision Relation Transformer for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2308.09472/) (2023)
- [Compositional Feature Augmentation for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2308.06712/) (2023)
- [Adaptive Fine-Grained Predicates Learning for Scene Graph Generation](https://mlsanity.com/paper/2207.04602/) (2022)
- [A Causal Adjustment Module for Debiasing Scene Graph Generation](https://mlsanity.com/paper/2503.17862/) (2025)
- [Unified Visual Relationship Detection with Vision and Language Models](https://mlsanity.com/paper/2303.08998/) (2023)
- [VLPrompt: Vision-Language Prompting for Panoptic Scene Graph Generation](https://mlsanity.com/paper/2311.16492/) (2023)
- [Adaptive Self-training Framework for Fine-grained Scene Graph Generation](https://mlsanity.com/paper/2401.09786/) (2024)
- [Towards Complex-query Referring Image Segmentation: A Novel Benchmark](https://mlsanity.com/paper/2309.17205/) (2023)
- [Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generation](https://mlsanity.com/paper/2409.10262/) (2024)
- [Environment-Invariant Curriculum Relation Learning for Fine-Grained Scene Graph Generation](https://mlsanity.com/paper/2308.03282/) (2023)
- [Haystack: A Panoptic Scene Graph Dataset to Evaluate Rare Predicate Classes](https://mlsanity.com/paper/2309.02286/) (2023)
- [Leveraging Predicate and Triplet Learning for Scene Graph Generation](https://mlsanity.com/paper/2406.02038/) (2024)
- [Domain-wise Invariant Learning for Panoptic Scene Graph Generation](https://mlsanity.com/paper/2310.05867/) (2023)
- [Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledge](https://mlsanity.com/paper/2311.12889/) (2023)
- [Scene-Graph ViT: End-to-End Open-Vocabulary Visual Relationship Detection](https://mlsanity.com/paper/2403.14270/) (2024)
- [RA-SGG: Retrieval-Augmented Scene Graph Generation Framework via Multi-Prototype Learning](https://mlsanity.com/paper/2412.12788/) (2024)
- [Skew Class-balanced Re-weighting for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2301.00351/) (2023)
- [AUG: A New Dataset and An Efficient Model for Aerial Image Urban Scene Graph Generation](https://mlsanity.com/paper/2404.07788/) (2024)
- [Fine-Grained is Too Coarse: A Novel Data-Centric Approach for Efficient Scene Graph Generation](https://mlsanity.com/paper/2305.18668/) (2023)
- [In Defense of Clip-based Video Relation Detection](https://mlsanity.com/paper/2307.08984/) (2023)

## Abstract (arXiv)

Scene graph generation (SGG) is designed to extract (subject, predicate, object) triplets in images. Recent works have made a steady progress on SGG, and provide useful tools for high-level vision and language understanding. However, due to the data distribution problems including long-tail distribution and semantic ambiguity, the predictions of current SGG models tend to collapse to several frequent but uninformative predicates (e.g., on, at), which limits practical application of these models in downstream tasks. To deal with the problems above, we propose a novel Internal and External Data Transfer (IETrans) method, which can be applied in a plug-and-play fashion and expanded to large SGG with 1,807 predicate classes. Our IETrans tries to relieve the data distribution problem by automatically creating an enhanced dataset that provides more sufficient and coherent annotations for all predicates. By training on the enhanced dataset, a Neural Motif model doubles the macro performance while maintaining competitive micro performance. The code and data are publicly available at https://github.com/waxnkw/IETrans-SGG.pytorch.