# Explanation-based Weakly-supervised Learning of Visual Relations with Graph Networks

arXiv: https://arxiv.org/abs/2006.09562 · HTML: https://mlsanity.com/paper/2006.09562/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 13 of 13)

- [Learning to Generate Scene Graph from Natural Language Supervision](https://mlsanity.com/paper/2109.02227/) (2021)
- [Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning](https://mlsanity.com/paper/2208.08165/) (2022)
- [Visual Distant Supervision for Scene Graph Generation](https://mlsanity.com/paper/2103.15365/) (2021)
- [Weakly-supervised HOI Detection via Prior-guided Bi-level Representation Learning](https://mlsanity.com/paper/2303.01313/) (2023)
- [Unpaired Image Captioning by Image-level Weakly-Supervised Visual Concept Recognition](https://mlsanity.com/paper/2203.03195/) (2022)
- [Exploiting CLIP for Zero-shot HOI Detection Requires Knowledge Distillation at Multiple Levels](https://mlsanity.com/paper/2309.05069/) (2023)
- [A Review of Human-Object Interaction Detection](https://mlsanity.com/paper/2408.10641/) (2024)
- [Quantitative Metrics for Evaluating Explanations of Video DeepFake Detectors](https://mlsanity.com/paper/2210.03683/) (2022)
- [Training image classifiers using Semi-Weak Label Data](https://mlsanity.com/paper/2103.10608/) (2021)
- [HICO-DET-SG and V-COCO-SG: New Data Splits for Evaluating the Systematic Generalization Performance of Human-Object Interaction Detection Models](https://mlsanity.com/paper/2305.09948/) (2023)
- [Graph Self-Supervised Learning for Endoscopic Image Matching](https://mlsanity.com/paper/2306.11141/) (2023)
- [FreeA: Human-object Interaction Detection using Free Annotation Labels](https://mlsanity.com/paper/2403.01840/) (2024)
- [RegFormer: Transferable Relational Grounding for Efficient Weakly-Supervised Human-Object Interaction Detection](https://mlsanity.com/paper/2604.00507/) (2026)

## Abstract (arXiv)

Visual relationship detection is fundamental for holistic image understanding. However, the localization and classification of (subject, predicate, object) triplets remain challenging tasks, due to the combinatorial explosion of possible relationships, their long-tailed distribution in natural images, and an expensive annotation process. This paper introduces a novel weakly-supervised method for visual relationship detection that relies on minimal image-level predicate labels. A graph neural network is trained to classify predicates in images from a graph representation of detected objects, implicitly encoding an inductive bias for pairwise relations. We then frame relationship detection as the explanation of such a predicate classifier, i.e. we obtain a complete relation by recovering the subject and object of a predicted predicate. We present results comparable to recent fully- and weakly-supervised methods on three diverse and challenging datasets: HICO-DET for human-object interaction, Visual Relationship Detection for generic object-to-object relations, and UnRel for unusual triplets; demonstrating robustness to non-comprehensive annotations and good few-shot generalization.