# Automatic Modeling of Social Concepts Evoked by Art Images as Multimodal Frames

arXiv: https://arxiv.org/abs/2110.07420 · HTML: https://mlsanity.com/paper/2110.07420/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 21 references that are in the map.
- First posted on arXiv 14 Oct 2021.

## Builds on (most cited first, 5 of 5)

- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [DeepSentiBank: Visual Sentiment Concept Classification with Deep Convolutional Neural Networks](https://mlsanity.com/paper/1410.8586/) (2014)
- [Improving Object Detection in Art Images Using Only Style Transfer](https://mlsanity.com/paper/2102.06529/) (2021)
- [Integrating Knowledge and Reasoning in Image Understanding](https://mlsanity.com/paper/1906.09954/) (2019)
- [Evaluation of Deep Learning on an Abstract Image Classification Dataset](https://mlsanity.com/paper/1708.07770/) (2017)

## Built on by (most cited first, 4 of 4)

- [Stitching Gaps: Fusing Situated Perceptual Knowledge with Vision Transformers for High-Level Image Classification](https://mlsanity.com/paper/2402.19339/) (2024)
- [Automating Iconclass: LLMs and RAG for Large-Scale Classification of Religious Woodcuts](https://mlsanity.com/paper/2510.19986/) (2025)
- [Situated Ground Truths: Enhancing Bias-Aware AI by Situating Data Labels with SituAnnotate](https://mlsanity.com/paper/2406.07583/) (2024)
- [A Pattern to Align Them All: Integrating Different Modalities to Define Multi-Modal Entities](https://mlsanity.com/paper/2410.13803/) (2024)

## Abstract (arXiv)

Social concepts referring to non-physical objects--such as revolution, violence, or friendship--are powerful tools to describe, index, and query the content of visual data, including ever-growing collections of art images from the Cultural Heritage (CH) field. While much progress has been made towards complete image understanding in computer vision, automatic detection of social concepts evoked by images is still a challenge. This is partly due to the well-known semantic gap problem, worsened for social concepts given their lack of unique physical features, and reliance on more unspecific features than concrete concepts. In this paper, we propose the translation of recent cognitive theories about social concept representation into a software approach to represent them as multimodal frames, by integrating multisensory data. Our method focuses on the extraction, analysis, and integration of multimodal features from visual art material tagged with the concepts of interest. We define a conceptual model and present a novel ontology for formally representing social concepts as multimodal frames. Taking the Tate Gallery's collection as an empirical basis, we experiment our method on a corpus of art images to provide a proof of concept of its potential. We discuss further directions of research, and provide all software, data sources, and results.