# One-shot Scene Graph Generation

arXiv: https://arxiv.org/abs/2202.10824 · HTML: https://mlsanity.com/paper/2202.10824/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 46 references that are in the map.
- First posted on arXiv 22 Feb 2022.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Embedding Entities and Relations for Learning and Inference in Knowledge Bases](https://mlsanity.com/paper/1412.6575/) (2014)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning](https://mlsanity.com/paper/1909.02151/) (2019)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [Scene Graph Generation from Objects, Phrases and Region Captions](https://mlsanity.com/paper/1707.09700/) (2017)
- [Visual Translation Embedding Network for Visual Relation Detection](https://mlsanity.com/paper/1702.08319/) (2017)
- [Music Gesture for Visual Sound Separation](https://mlsanity.com/paper/2004.09476/) (2020)
- [Scene Graph Generation with External Knowledge and Image Reconstruction](https://mlsanity.com/paper/1904.00560/) (2019)
- [Foley Music: Learning to Generate Music from Videos](https://mlsanity.com/paper/2007.10984/) (2020)
- [Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generation](https://mlsanity.com/paper/1806.11538/) (2018)
- [Pixels to Graphs by Associative Embedding](https://mlsanity.com/paper/1706.07365/) (2017)
- [Counterfactual Critic Multi-Agent Training for Scene Graph Generation](https://mlsanity.com/paper/1812.02347/) (2018)
- [Zoom-Net: Mining Deep Feature Interactions for Visual Relationship Recognition](https://mlsanity.com/paper/1807.04979/) (2018)
- [Self-Supervised Video Hashing with Hierarchical Binary Auto-encoder](https://mlsanity.com/paper/1802.02305/) (2018)
- [Universal Weighting Metric Learning for Cross-Modal Matching](https://mlsanity.com/paper/2010.03403/) (2020)
- [Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning](https://mlsanity.com/paper/1706.01231/) (2017)

## Built on by (most cited first, 8 of 8)

- [Prototype-based Embedding Network for Scene Graph Generation](https://mlsanity.com/paper/2303.07096/) (2023)
- [Fine-Grained Predicates Learning for Scene Graph Generation](https://mlsanity.com/paper/2204.02597/) (2022)
- [Dual-branch Hybrid Learning Network for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2207.07913/) (2022)
- [Explore Contextual Information for 3D Scene Graph Generation](https://mlsanity.com/paper/2210.06240/) (2022)
- [Decomposed Prototype Learning for Few-Shot Scene Graph Generation](https://mlsanity.com/paper/2303.10863/) (2023)
- [HGOE: Hybrid External and Internal Graph Outlier Exposure for Graph Out-of-Distribution Detection](https://mlsanity.com/paper/2407.21742/) (2024)
- [Informative Scene Graph Generation via Debiasing](https://mlsanity.com/paper/2308.05286/) (2023)
- [PointViG: A Lightweight GNN-based Model for Efficient Point Cloud Analysis](https://mlsanity.com/paper/2407.00921/) (2024)

## Abstract (arXiv)

As a structured representation of the image content, the visual scene graph (visual relationship) acts as a bridge between computer vision and natural language processing. Existing models on the scene graph generation task notoriously require tens or hundreds of labeled samples. By contrast, human beings can learn visual relationships from a few or even one example. Inspired by this, we design a task named One-Shot Scene Graph Generation, where each relationship triplet (e.g., "dog-has-head") comes from only one labeled example. The key insight is that rather than learning from scratch, one can utilize rich prior knowledge. In this paper, we propose Multiple Structured Knowledge (Relational Knowledge and Commonsense Knowledge) for the one-shot scene graph generation task. Specifically, the Relational Knowledge represents the prior knowledge of relationships between entities extracted from the visual content, e.g., the visual relationships "standing in", "sitting in", and "lying in" may exist between "dog" and "yard", while the Commonsense Knowledge encodes "sense-making" knowledge like "dog can guard yard". By organizing these two kinds of knowledge in a graph structure, Graph Convolution Networks (GCNs) are used to extract knowledge-embedded semantic features of the entities. Besides, instead of extracting isolated visual features from each entity generated by Faster R-CNN, we utilize an Instance Relation Transformer encoder to fully explore their context information. Based on a constructed one-shot dataset, the experimental results show that our method significantly outperforms existing state-of-the-art methods by a large margin. Ablation studies also verify the effectiveness of the Instance Relation Transformer encoder and the Multiple Structured Knowledge.