# Symbolic image detection using scene and knowledge graphs

arXiv: https://arxiv.org/abs/2206.04863 · HTML: https://mlsanity.com/paper/2206.04863/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 21 references that are in the map.
- First posted on arXiv 10 Jun 2022.

## Builds on (most cited first, 7 of 7)

- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [The THUMOS Challenge on Action Recognition for Videos "in the Wild"](https://mlsanity.com/paper/1604.06182/) (2016)
- [Automatic Understanding of Image and Video Advertisements](https://mlsanity.com/paper/1707.03067/) (2017)
- [ADVISE: Symbolism and External Knowledge for Decoding Advertisements](https://mlsanity.com/paper/1711.06666/) (2017)
- [Understanding the Role of Scene Graphs in Visual Question Answering](https://mlsanity.com/paper/2101.05479/) (2021)
- [GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answering](https://mlsanity.com/paper/2104.10283/) (2021)

## Built on by (most cited first, 3 of 3)

- [Visual Objectification in Films: Towards a New AI Task for Video Interpretation](https://mlsanity.com/paper/2401.13296/) (2024)
- [Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understanding](https://mlsanity.com/paper/2508.20765/) (2025)
- [Stitching Gaps: Fusing Situated Perceptual Knowledge with Vision Transformers for High-Level Image Classification](https://mlsanity.com/paper/2402.19339/) (2024)

## Abstract (arXiv)

Sometimes the meaning conveyed by images goes beyond the list of objects they contain; instead, images may express a powerful message to affect the viewers' minds. Inferring this message requires reasoning about the relationships between the objects, and general common-sense knowledge about the components. In this paper, we use a scene graph, a graph representation of an image, to capture visual components. In addition, we generate a knowledge graph using facts extracted from ConceptNet to reason about objects and attributes. To detect the symbols, we propose a neural network framework named SKG-Sym. The framework first generates the representations of the scene graph of the image and its knowledge graph using Graph Convolution Network. The framework then fuses the representations and uses an MLP to classify them. We extend the network further to use an attention mechanism which learn the importance of the graph representations. We evaluate our methods on a dataset of advertisements, and compare it with baseline symbolism classification methods (ResNet and VGG). Results show that our methods outperform ResNet in terms of F-score and the attention-based mechanism is competitive with VGG while it has much lower model complexity.