# Graphical Contrastive Losses for Scene Graph Parsing

arXiv: https://arxiv.org/abs/1903.02728 · HTML: https://mlsanity.com/paper/1903.02728/

## Key facts

- Cited by 100 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Mar 2019.

## Built on by (most cited first, 30 of 100)

- [Energy-Based Learning for Scene Graph Generation](https://mlsanity.com/paper/2103.02221/) (2021)
- [Spatial-Temporal Transformer for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2107.12309/) (2021)
- [RelTR: Relation Transformer for Scene Graph Generation](https://mlsanity.com/paper/2201.11460/) (2022)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Learning to Generate Scene Graph from Natural Language Supervision](https://mlsanity.com/paper/2109.02227/) (2021)
- [Target Adaptive Context Aggregation for Video Scene Graph Generation](https://mlsanity.com/paper/2108.08121/) (2021)
- [FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing](https://mlsanity.com/paper/2305.17497/) (2023)
- [Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detection](https://mlsanity.com/paper/2206.06291/) (2022)
- [EGTR: Extracting Graph from Transformer for Scene Graph Generation](https://mlsanity.com/paper/2404.02072/) (2024)
- [RLIPv2: Fast Scaling of Relational Language-Image Pre-training](https://mlsanity.com/paper/2308.09351/) (2023)
- [Structured Sparse R-CNN for Direct Scene Graph Generation](https://mlsanity.com/paper/2106.10815/) (2021)
- [HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention](https://mlsanity.com/paper/2303.02995/) (2023)
- [Unbiased Scene Graph Generation in Videos](https://mlsanity.com/paper/2304.00733/) (2023)
- [Unified Graph Structured Models for Video Understanding](https://mlsanity.com/paper/2103.15662/) (2021)
- [Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality](https://mlsanity.com/paper/2305.13812/) (2023)
- [Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queries](https://mlsanity.com/paper/2105.02170/) (2021)
- [Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphs](https://mlsanity.com/paper/2112.09828/) (2021)
- [Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention](https://mlsanity.com/paper/2311.10988/) (2023)
- [RU-Net: Regularized Unrolling Network for Scene Graph Generation](https://mlsanity.com/paper/2205.01297/) (2022)
- [Visual Distant Supervision for Scene Graph Generation](https://mlsanity.com/paper/2103.15365/) (2021)
- [Iterative Scene Graph Generation](https://mlsanity.com/paper/2207.13440/) (2022)
- [OED: Towards One-stage End-to-End Dynamic Scene Graph Generation](https://mlsanity.com/paper/2405.16925/) (2024)
- [HL-Net: Heterophily Learning Network for Scene Graph Generation](https://mlsanity.com/paper/2205.01316/) (2022)
- [Iterative Scene Graph Generation with Generative Transformers](https://mlsanity.com/paper/2211.16636/) (2022)
- [Not All Relations are Equal: Mining Informative Labels for Scene Graph Generation](https://mlsanity.com/paper/2111.13517/) (2021)
- [Meta Spatio-Temporal Debiasing for Video Scene Graph Generation](https://mlsanity.com/paper/2207.11441/) (2022)
- [BGT-Net: Bidirectional GRU Transformer Network for Scene Graph Generation](https://mlsanity.com/paper/2109.05346/) (2021)
- [Unbiased Heterogeneous Scene Graph Generation with Relation-aware Message Passing Neural Network](https://mlsanity.com/paper/2212.00443/) (2022)
- [Pair then Relation: Pair-Net for Panoptic Scene Graph Generation](https://mlsanity.com/paper/2307.08699/) (2023)

## Abstract (arXiv)

Most scene graph parsers use a two-stage pipeline to detect visual relationships: the first stage detects entities, and the second predicts the predicate for each entity pair using a softmax distribution. We find that such pipelines, trained with only a cross entropy loss over predicate classes, suffer from two common errors. The first, Entity Instance Confusion, occurs when the model confuses multiple instances of the same type of entity (e.g. multiple cups). The second, Proximal Relationship Ambiguity, arises when multiple subject-predicate-object triplets appear in close proximity with the same predicate, and the model struggles to infer the correct subject-object pairings (e.g. mis-pairing musicians and their instruments). We propose a set of contrastive loss formulations that specifically target these types of errors within the scene graph parsing problem, collectively termed the Graphical Contrastive Losses. These losses explicitly force the model to disambiguate related and unrelated instances through margin constraints specific to each type of confusion. We further construct a relationship detector, called RelDN, using the aforementioned pipeline to demonstrate the efficacy of our proposed losses. Our model outperforms the winning method of the OpenImages Relationship Detection Challenge by 4.7% (16.5% relative) on the test set. We also show improved results over the best previous methods on the Visual Genome and Visual Relationship Detection datasets.