# Image Semantic Relation Generation

arXiv: https://arxiv.org/abs/2210.11253 · HTML: https://mlsanity.com/paper/2210.11253/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Autoregressive Entity Retrieval](https://mlsanity.com/paper/2010.00904/) (2020)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Scene Graph Generation from Objects, Phrases and Region Captions](https://mlsanity.com/paper/1707.09700/) (2017)
- [Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge](https://mlsanity.com/paper/1609.06647/) (2016)
- [Panoptic Scene Graph Generation](https://mlsanity.com/paper/2207.11247/) (2022)
- [A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP](https://mlsanity.com/paper/2108.13002/) (2021)

## Abstract (arXiv)

Scene graphs provide structured semantic understanding beyond images. For downstream tasks, such as image retrieval, visual question answering, visual relationship detection, and even autonomous vehicle technology, scene graphs can not only distil complex image information but also correct the bias of visual models using semantic-level relations, which has broad application prospects. However, the heavy labour cost of constructing graph annotations may hinder the application of PSG in practical scenarios. Inspired by the observation that people usually identify the subject and object first and then determine the relationship between them, we proposed to decouple the scene graphs generation task into two sub-tasks: 1) an image segmentation task to pick up the qualified objects. 2) a restricted auto-regressive text generation task to generate the relation between given objects. Therefore, in this work, we introduce image semantic relation generation (ISRG), a simple but effective image-to-text model, which achieved 31 points on the OpenPSG dataset and outperforms strong baselines respectively by 16 points (ResNet-50) and 5 points (CLIP).