# Prototype-based Embedding Network for Scene Graph Generation

arXiv: https://arxiv.org/abs/2303.07096 · HTML: https://mlsanity.com/paper/2303.07096/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 45 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](https://mlsanity.com/paper/2003.12962/) (2020)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2104.00308/) (2021)
- [Visual Translation Embedding Network for Visual Relation Detection](https://mlsanity.com/paper/1702.08319/) (2017)
- [Energy-Based Learning for Scene Graph Generation](https://mlsanity.com/paper/2103.02221/) (2021)
- [SGTR: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2112.12970/) (2021)
- [PCPL: Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2009.00893/) (2020)
- [Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2203.09811/) (2022)
- [CogTree: Cognition Tree Loss for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2009.07526/) (2020)
- [The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generation](https://mlsanity.com/paper/2206.03014/) (2022)
- [From General to Specific: Informative Scene Graph Generation via Balance Adjustment](https://mlsanity.com/paper/2108.13129/) (2021)
- [Structured Sparse R-CNN for Direct Scene Graph Generation](https://mlsanity.com/paper/2106.10815/) (2021)
- [Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generation](https://mlsanity.com/paper/2103.05271/) (2021)
- [Structured Query-Based Image Retrieval Using Scene Graphs](https://mlsanity.com/paper/2005.06653/) (2020)
- [RU-Net: Regularized Unrolling Network for Scene Graph Generation](https://mlsanity.com/paper/2205.01297/) (2022)
- [HL-Net: Heterophily Learning Network for Scene Graph Generation](https://mlsanity.com/paper/2205.01316/) (2022)
- [Adaptive Fine-Grained Predicates Learning for Scene Graph Generation](https://mlsanity.com/paper/2207.04602/) (2022)

## Built on by (most cited first, 30 of 37)

- [From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models](https://mlsanity.com/paper/2404.00906/) (2024)
- [STAR: A First-Ever Dataset and A Large-Scale Benchmark for Scene Graph Generation in Large-Size Satellite Imagery](https://mlsanity.com/paper/2406.09410/) (2024)
- [HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding](https://mlsanity.com/paper/2312.03050/) (2023)
- [Scene Graph Generation with Role-Playing Large Language Models](https://mlsanity.com/paper/2410.15364/) (2024)
- [HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation](https://mlsanity.com/paper/2403.12033/) (2024)
- [A Causal Adjustment Module for Debiasing Scene Graph Generation](https://mlsanity.com/paper/2503.17862/) (2025)
- [Adaptive Self-training Framework for Fine-grained Scene Graph Generation](https://mlsanity.com/paper/2401.09786/) (2024)
- [DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation](https://mlsanity.com/paper/2403.14886/) (2024)
- [Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generation](https://mlsanity.com/paper/2409.10262/) (2024)
- [Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering](https://mlsanity.com/paper/2412.20927/) (2024)
- [Leveraging Predicate and Triplet Learning for Scene Graph Generation](https://mlsanity.com/paper/2406.02038/) (2024)
- [Stellar: Systematic Evaluation of Human-Centric Personalized Text-to-Image Methods](https://mlsanity.com/paper/2312.06116/) (2023)
- [Towards Lifelong Scene Graph Generation with Knowledge-ware In-context Prompt Learning](https://mlsanity.com/paper/2401.14626/) (2024)
- [CREPE: Learnable Prompting With CLIP Improves Visual Relationship Prediction](https://mlsanity.com/paper/2307.04838/) (2023)
- [Improving Scene Graph Generation with Superpixel-Based Interaction Learning](https://mlsanity.com/paper/2308.02339/) (2023)
- [Joint Generative Modeling of Grounded Scene Graphs and Images via Diffusion Models](https://mlsanity.com/paper/2401.01130/) (2024)
- [Scene Graph Generation Strategy with Co-occurrence Knowledge and Learnable Term Frequency](https://mlsanity.com/paper/2405.12648/) (2024)
- [Semantic Diversity-aware Prototype-based Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2407.15396/) (2024)
- [Generalized Unbiased Scene Graph Generation](https://mlsanity.com/paper/2308.04802/) (2023)
- [REACT: Real-time Efficiency and Accuracy Compromise for Tradeoffs in Scene Graph Generation](https://mlsanity.com/paper/2405.16116/) (2024)
- [ReCon1M:A Large-scale Benchmark Dataset for Relation Comprehension in Remote Sensing Imagery](https://mlsanity.com/paper/2406.06028/) (2024)
- [Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution Can Improve Robust Scene Graph Generation](https://mlsanity.com/paper/2504.12606/) (2025)
- [Informative Scene Graph Generation via Debiasing](https://mlsanity.com/paper/2308.05286/) (2023)
- [A Reverse Causal Framework to Mitigate Spurious Correlations for Debiasing Scene Graph Generation](https://mlsanity.com/paper/2505.23451/) (2025)
- [Salience-SGG: Enhancing Unbiased Scene Graph Generation with Iterative Salience Estimation](https://mlsanity.com/paper/2601.08728/) (2026)
- [REACT++: Efficient Cross-Attention for Real-Time Scene Graph Generation](https://mlsanity.com/paper/2603.06386/) (2026)
- [Local-Global Information Interaction Debiasing for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2308.05274/) (2023)
- [Head-Tail Cooperative Learning Network for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2308.12048/) (2023)
- [Unbiased Scene Graph Generation by Type-Aware Message Passing on Heterogeneous and Dual Graphs](https://mlsanity.com/paper/2411.13287/) (2024)
- [ART: Adaptive Relation Tuning for Generalized Relation Prediction](https://mlsanity.com/paper/2507.23543/) (2025)

## Abstract (arXiv)

Current Scene Graph Generation (SGG) methods explore contextual information to predict relationships among entity pairs. However, due to the diverse visual appearance of numerous possible subject-object combinations, there is a large intra-class variation within each predicate category, e.g., "man-eating-pizza, giraffe-eating-leaf", and the severe inter-class similarity between different classes, e.g., "man-holding-plate, man-eating-pizza", in model's latent space. The above challenges prevent current SGG methods from acquiring robust features for reliable relation prediction. In this paper, we claim that the predicate's category-inherent semantics can serve as class-wise prototypes in the semantic space for relieving the challenges. To the end, we propose the Prototype-based Embedding Network (PE-Net), which models entities/predicates with prototype-aligned compact and distinctive representations and thereby establishes matching between entity pairs and predicates in a common embedding space for relation recognition. Moreover, Prototype-guided Learning (PL) is introduced to help PE-Net efficiently learn such entitypredicate matching, and Prototype Regularization (PR) is devised to relieve the ambiguous entity-predicate matching caused by the predicate's semantic overlap. Extensive experiments demonstrate that our method gains superior relation recognition capability on SGG, achieving new state-of-the-art performances on both Visual Genome and Open Images datasets.