# Generative Compositional Augmentations for Scene Graph Prediction

arXiv: https://arxiv.org/abs/2007.05756 · HTML: https://mlsanity.com/paper/2007.05756/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jul 2020.

## Built on by (most cited first, 14 of 14)

- [SGTR: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2112.12970/) (2021)
- [Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning](https://mlsanity.com/paper/2208.08165/) (2022)
- [Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World](https://mlsanity.com/paper/2303.13233/) (2023)
- [Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention](https://mlsanity.com/paper/2311.10988/) (2023)
- [Devil's on the Edges: Selective Quad Attention for Scene Graph Generation](https://mlsanity.com/paper/2304.03495/) (2023)
- [Compositional Feature Augmentation for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2308.06712/) (2023)
- [Generalized Unbiased Scene Graph Generation](https://mlsanity.com/paper/2308.04802/) (2023)
- [Free-Form Composition Networks for Egocentric Action Recognition](https://mlsanity.com/paper/2307.06527/) (2023)
- [Informative Scene Graph Generation via Debiasing](https://mlsanity.com/paper/2308.05286/) (2023)
- [Zero-Shot Scene Graph Generation via Triplet Calibration and Reduction](https://mlsanity.com/paper/2309.03542/) (2023)
- [SGTR+: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2401.12835/) (2024)
- [SGDraw: Scene Graph Drawing Interface Using Object-Oriented Representation](https://mlsanity.com/paper/2211.16697/) (2022)
- [Towards Unseen Triples: Effective Text-Image-joint Learning for Scene Graph Generation](https://mlsanity.com/paper/2306.13420/) (2023)
- [Modeling Scientific Experiment Scenes: Dataset and Model](https://mlsanity.com/paper/2608.02892/) (2026)

## Abstract (arXiv)

Inferring objects and their relationships from an image in the form of a scene graph is useful in many applications at the intersection of vision and language. We consider a challenging problem of compositional generalization that emerges in this task due to a long tail data distribution. Current scene graph generation models are trained on a tiny fraction of the distribution corresponding to the most frequent compositions, e.g. <cup, on, table>. However, test images might contain zero- and few-shot compositions of objects and relationships, e.g. <cup, on, surfboard>. Despite each of the object categories and the predicate (e.g. 'on') being frequent in the training data, the models often fail to properly understand such unseen or rare compositions. To improve generalization, it is natural to attempt increasing the diversity of the training distribution. However, in the graph domain this is non-trivial. To that end, we propose a method to synthesize rare yet plausible scene graphs by perturbing real ones. We then propose and empirically study a model based on conditional generative adversarial networks (GANs) that allows us to generate visual features of perturbed scene graphs and learn from them in a joint fashion. When evaluated on the Visual Genome dataset, our approach yields marginal, but consistent improvements in zero- and few-shot metrics. We analyze the limitations of our approach indicating promising directions for future research.