# Probabilistic Modeling of Semantic Ambiguity for Scene Graph Generation

arXiv: https://arxiv.org/abs/2103.05271 · HTML: https://mlsanity.com/paper/2103.05271/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 44 references that are in the map.
- First posted on arXiv 9 Mar 2021.

## Builds on (most cited first, 30 of 33)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders](https://mlsanity.com/paper/1703.10960/) (2017)
- [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](https://mlsanity.com/paper/2003.12962/) (2020)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [Scene Graph Generation from Objects, Phrases and Region Captions](https://mlsanity.com/paper/1707.09700/) (2017)
- [Data Uncertainty Learning in Face Recognition](https://mlsanity.com/paper/2003.11339/) (2020)
- [Visual Translation Embedding Network for Visual Relation Detection](https://mlsanity.com/paper/1702.08319/) (2017)
- [Scene Graph Generation with External Knowledge and Image Reconstruction](https://mlsanity.com/paper/1904.00560/) (2019)
- [Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Images](https://mlsanity.com/paper/1703.10660/) (2017)
- [Word Representations via Gaussian Embedding](https://mlsanity.com/paper/1412.6623/) (2014)
- [Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation](https://mlsanity.com/paper/1806.11538/) (2018)
- [Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles](https://mlsanity.com/paper/1606.07839/) (2016)
- [Towards Diverse and Natural Image Descriptions via a Conditional GAN](https://mlsanity.com/paper/1703.06029/) (2017)
- [Comprehensive Image Captioning via Scene Graph Decomposition](https://mlsanity.com/paper/2007.11731/) (2020)
- [DP-GAN: Diversity-Promoting Generative Adversarial Network for Generating Informative and Diversified Text](https://mlsanity.com/paper/1802.01345/) (2018)
- [Learning to Count Objects in Natural Images for Visual Question Answering](https://mlsanity.com/paper/1802.05766/) (2018)
- [Confident Multiple Choice Learning](https://mlsanity.com/paper/1706.03475/) (2017)
- [Jointly Optimizing Diversity and Relevance in Neural Response Generation](https://mlsanity.com/paper/1902.11205/) (2019)
- [An Empirical Study on Leveraging Scene Graphs for Visual Question Answering](https://mlsanity.com/paper/1907.12133/) (2019)
- [Towards Context-aware Interaction Recognition](https://mlsanity.com/paper/1703.06246/) (2017)

## Built on by (most cited first, 30 of 35)

- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [SGTR: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2112.12970/) (2021)
- [MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model](https://mlsanity.com/paper/2210.05335/) (2022)
- [Prototype-based Embedding Network for Scene Graph Generation](https://mlsanity.com/paper/2303.07096/) (2023)
- [RU-Net: Regularized Unrolling Network for Scene Graph Generation](https://mlsanity.com/paper/2205.01297/) (2022)
- [HL-Net: Heterophily Learning Network for Scene Graph Generation](https://mlsanity.com/paper/2205.01316/) (2022)
- [Iterative Scene Graph Generation with Generative Transformers](https://mlsanity.com/paper/2211.16636/) (2022)
- [Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation](https://mlsanity.com/paper/2111.13517/) (2021)
- [Meta Spatio-Temporal Debiasing for Video Scene Graph Generation](https://mlsanity.com/paper/2207.11441/) (2022)
- [Probabilistic Prompt Learning for Dense Prediction](https://mlsanity.com/paper/2304.00779/) (2023)
- [Pair then Relation: Pair-Net for Panoptic Scene Graph Generation](https://mlsanity.com/paper/2307.08699/) (2023)
- [Adaptive Fine-Grained Predicates Learning for Scene Graph Generation](https://mlsanity.com/paper/2207.04602/) (2022)
- [Automated data processing and feature engineering for deep learning and big data applications: a survey](https://mlsanity.com/paper/2403.11395/) (2024)
- [Improving Visual Recognition with Hyperbolical Visual Hierarchy Mapping](https://mlsanity.com/paper/2404.00974/) (2024)
- [DuSSS: Dual Semantic Similarity-Supervised Vision-Language Model for Semi-Supervised Medical Image Segmentation](https://mlsanity.com/paper/2412.12492/) (2024)
- [Probabilistic Debiasing of Scene Graphs](https://mlsanity.com/paper/2211.06444/) (2022)
- [ProbVLM: Probabilistic Adapter for Frozen Vision-Language Models](https://mlsanity.com/paper/2307.00398/) (2023)
- [Conformal Prediction and MLLM aided Uncertainty Quantification in Scene Graph Generation](https://mlsanity.com/paper/2503.13947/) (2025)
- [Skew Class-balanced Re-weighting for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2301.00351/) (2023)
- [AUG: A New Dataset and An Efficient Model for Aerial Image Urban Scene Graph Generation](https://mlsanity.com/paper/2404.07788/) (2024)
- [Explore Contextual Information for 3D Scene Graph Generation](https://mlsanity.com/paper/2210.06240/) (2022)
- [CREPE: Learnable Prompting With CLIP Improves Visual Relationship Prediction](https://mlsanity.com/paper/2307.04838/) (2023)
- [Context-aware Mixture-of-Experts for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2208.07109/) (2022)
- [Can Vision Language Models Learn from Visual Demonstrations of Ambiguous Spatial Reasoning?](https://mlsanity.com/paper/2409.17080/) (2024)
- [Semantic Scene Graph Generation Based on an Edge Dual Scene Graph and Message Passing Neural Network](https://mlsanity.com/paper/2311.01192/) (2023)
- [IMPACT-CYCLE: A Contract-Based Multi-Agent System for Claim-Level Supervisory Correction of Long-Video Semantic Memory](https://mlsanity.com/paper/2604.20136/) (2026)
- [Self-Supervised Learning for Visual Relationship Detection through Masked Bounding Box Reconstruction](https://mlsanity.com/paper/2311.04834/) (2023)
- [SGTR+: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2401.12835/) (2024)
- [Predicate Debiasing in Vision-Language Models Integration for Scene Graph Generation Enhancement](https://mlsanity.com/paper/2403.16184/) (2024)
- [PARIC: Probabilistic Attention Regularization for Language Guided Image Classification from Pre-trained Vison Language Models](https://mlsanity.com/paper/2503.11360/) (2025)

## Abstract (arXiv)

To generate "accurate" scene graphs, almost all existing methods predict pairwise relationships in a deterministic manner. However, we argue that visual relationships are often semantically ambiguous. Specifically, inspired by linguistic knowledge, we classify the ambiguity into three types: Synonymy Ambiguity, Hyponymy Ambiguity, and Multi-view Ambiguity. The ambiguity naturally leads to the issue of implicit multi-label, motivating the need for diverse predictions. In this work, we propose a novel plug-and-play Probabilistic Uncertainty Modeling (PUM) module. It models each union region as a Gaussian distribution, whose variance measures the uncertainty of the corresponding visual content. Compared to the conventional deterministic methods, such uncertainty modeling brings stochasticity of feature representation, which naturally enables diverse predictions. As a byproduct, PUM also manages to cover more fine-grained relationships and thus alleviates the issue of bias towards frequent relationships. Extensive experiments on the large-scale Visual Genome benchmark show that combining PUM with newly proposed ResCAGCN can achieve state-of-the-art performances, especially under the mean recall metric. Furthermore, we prove the universal effectiveness of PUM by plugging it into some existing models and provide insightful analysis of its ability to generate diverse yet plausible visual relationships.