# Scene Graph Prediction with Limited Labels

arXiv: https://arxiv.org/abs/1904.11622 · HTML: https://mlsanity.com/paper/1904.11622/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2019.

## Built on by (most cited first, 30 of 32)

- [AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning](https://mlsanity.com/paper/2103.16002/) (2021)
- [Spatial-Temporal Transformer for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2107.12309/) (2021)
- [WRENCH: A Comprehensive Benchmark for Weak Supervision](https://mlsanity.com/paper/2109.11377/) (2021)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Recovering the Unbiased Scene Graphs from the Biased Ones](https://mlsanity.com/paper/2107.02112/) (2021)
- [A Survey on Programmatic Weak Supervision](https://mlsanity.com/paper/2202.05433/) (2022)
- [Learning of Visual Relations: The Devil is in the Tails](https://mlsanity.com/paper/2108.09668/) (2021)
- [Fully Convolutional Scene Graph Generation](https://mlsanity.com/paper/2103.16083/) (2021)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models](https://mlsanity.com/paper/2404.00906/) (2024)
- [Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World](https://mlsanity.com/paper/2303.13233/) (2023)
- [Visual Distant Supervision for Scene Graph Generation](https://mlsanity.com/paper/2103.15365/) (2021)
- [Category-Level Multi-Part Multi-Joint 3D Shape Assembly](https://mlsanity.com/paper/2303.06163/) (2023)
- [DP-SSL: Towards Robust Semi-supervised Learning with A Few Labeled Samples](https://mlsanity.com/paper/2110.13740/) (2021)
- [RICE: Refining Instance Masks in Cluttered Environments with Graph Neural Networks](https://mlsanity.com/paper/2106.15711/) (2021)
- [Learning from Multiple Noisy Partial Labelers](https://mlsanity.com/paper/2106.04530/) (2021)
- [Label Semantic Knowledge Distillation for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2208.03763/) (2022)
- [ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation](https://mlsanity.com/paper/2311.13258/) (2023)
- [Active WeaSuL: Improving Weak Supervision with Active Learning](https://mlsanity.com/paper/2104.14847/) (2021)
- [HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations](https://mlsanity.com/paper/2106.00162/) (2021)
- [Rethinking the Evaluation of Unbiased Scene Graph Generation](https://mlsanity.com/paper/2208.01909/) (2022)
- [Knowledge-augmented Few-shot Visual Relation Detection](https://mlsanity.com/paper/2303.05342/) (2023)
- [Analytical Survey of Learning with Low-Resource Data: From Analysis to Investigation](https://mlsanity.com/paper/2510.08962/) (2025)
- [CREPE: Learnable Prompting With CLIP Improves Visual Relationship Prediction](https://mlsanity.com/paper/2307.04838/) (2023)
- [Situational Scene Graph for Structured Human-centric Situation Understanding](https://mlsanity.com/paper/2410.22829/) (2024)
- [Binary Classification with Positive Labeling Sources](https://mlsanity.com/paper/2208.01704/) (2022)
- [Search for Concepts: Discovering Visual Concepts Using Direct Optimization](https://mlsanity.com/paper/2210.14808/) (2022)
- [A Survey of Learning on Small Data: Generalization, Optimization, and Challenge](https://mlsanity.com/paper/2207.14443/) (2022)
- [An Adaptive Method for Weak Supervision with Drifting Data](https://mlsanity.com/paper/2306.01658/) (2023)
- [Map Prediction and Generative Entropy for Multi-Agent Exploration](https://mlsanity.com/paper/2501.13189/) (2025)

## Abstract (arXiv)

Visual knowledge bases such as Visual Genome power numerous applications in computer vision, including visual question answering and captioning, but suffer from sparse, incomplete relationships. All scene graph models to date are limited to training on a small set of visual relationships that have thousands of training labels each. Hiring human annotators is expensive, and using textual knowledge base completion methods are incompatible with visual data. In this paper, we introduce a semi-supervised method that assigns probabilistic relationship labels to a large number of unlabeled images using few labeled examples. We analyze visual relationships to suggest two types of image-agnostic features that are used to generate noisy heuristics, whose outputs are aggregated using a factor graph-based generative model. With as few as 10 labeled examples per relationship, the generative model creates enough training data to train any existing state-of-the-art scene graph model. We demonstrate that our method outperforms all baseline approaches on scene graph prediction by 5.16 recall@100 for PREDCLS. In our limited label setting, we define a complexity metric for relationships that serves as an indicator (R^2 = 0.778) for conditions under which our method succeeds over transfer learning, the de-facto approach for training with limited labels.