Extracting associations and meanings of objects depicted in artworks through bi-modal deep networks
Not on the map. Why?
Abstract (arXiv)
We present a novel bi-modal system based on deep networks to address the problem of learning associations and simple meanings of objects depicted in "authored" images, such as fine art paintings and drawings. Our overall system processes both the images and associated texts in order to learn associations between images of individual objects, their identities and the abstract meanings they signify. Unlike past deep nets that describe depicted objects and infer predicates, our system identifies meaning-bearing objects ("signifiers") and their associations ("signifieds") as well as basic overall meanings for target artworks. Our system had precision of 48% and recall of 78% with an F1 metric of 0.6 on a curated set of Dutch vanitas paintings, a genre celebrated for its concentration on conveying a meaning of great import at the time of their execution. We developed and tested our system on fine art paintings but our general methods can be applied to other authored images.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 7 of its 15 references that are in the map. First posted on arXiv 14 Mar 2022.
Builds on
7 of 15 references.
References · 7
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2017Mask R-CNNcited by 7,277
- 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,487
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
- 2019Simple BERT Models for Relation Extraction and Semantic Role Labelingcited by 154
- 2021Computational identification of significant actors in paintings through symbols and attributescited by 3
Built on by
Citing papers per year: 2023: 1.
1 paper.