# How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval

arXiv: https://arxiv.org/abs/1810.09617 · HTML: https://mlsanity.com/paper/1810.09617/

## Key facts

- Cited by 74 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2018.

## Built on by (most cited first, 30 of 74)

- [Measuring Style Similarity in Diffusion Models](https://mlsanity.com/paper/2404.01292/) (2024)
- [ArtEmis: Affective Language for Visual Art](https://mlsanity.com/paper/2101.07396/) (2021)
- [Human-Art: A Versatile Human-Centric Dataset Bridging Natural and Artificial Scenes](https://mlsanity.com/paper/2303.02760/) (2023)
- [The ArtBench Dataset: Benchmarking Generative Models with Artworks](https://mlsanity.com/paper/2206.11404/) (2022)
- [Understanding and Creating Art with AI: Review and Outlook](https://mlsanity.com/paper/2102.09109/) (2021)
- [The Met Dataset: Instance-level Recognition for Artworks](https://mlsanity.com/paper/2202.01747/) (2022)
- [Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation](https://mlsanity.com/paper/2109.05743/) (2021)
- [LMEye: An Interactive Perception Network for Large Language Models](https://mlsanity.com/paper/2305.03701/) (2023)
- [Q: How to Specialize Large Vision-Language Models to Data-Scarce VQA Tasks? A: Self-Train on Unlabeled Images!](https://mlsanity.com/paper/2306.03932/) (2023)
- [An overview of domain-specific foundation model: key technologies, applications and challenges](https://mlsanity.com/paper/2409.04267/) (2024)
- [Not Only Generative Art: Stable Diffusion for Content-Style Disentanglement in Art Analysis](https://mlsanity.com/paper/2304.10278/) (2023)
- [Conditional LoRA Parameter Generation](https://mlsanity.com/paper/2408.01415/) (2024)
- [Iconographic Image Captioning for Artworks](https://mlsanity.com/paper/2102.03942/) (2021)
- [OpenMAG: A Comprehensive Benchmark for Multimodal-Attributed Graph](https://mlsanity.com/paper/2602.05576/) (2026)
- [StyleBabel: Artistic Style Tagging and Captioning](https://mlsanity.com/paper/2203.05321/) (2022)
- [GCNBoost: Artwork Classification by Label Propagation through a Knowledge Graph](https://mlsanity.com/paper/2105.11852/) (2021)
- [A survey on knowledge-enhanced multimodal learning](https://mlsanity.com/paper/2211.12328/) (2022)
- [Simple Disentanglement of Style and Content in Visual Representations](https://mlsanity.com/paper/2302.09795/) (2023)
- [ARTxAI: Explainable Artificial Intelligence Curates Deep Representation Learning for Artistic Images using Fuzzy Techniques](https://mlsanity.com/paper/2308.15284/) (2023)
- [Graph Neural Networks for Knowledge Enhanced Visual Representation of Paintings](https://mlsanity.com/paper/2105.08190/) (2021)
- [DEArt: Dataset of European Art](https://mlsanity.com/paper/2211.01226/) (2022)
- [Prototype-based Dataset Comparison](https://mlsanity.com/paper/2309.02401/) (2023)
- [ArtRAG: Retrieval-Augmented Generation with Structured Context for Visual Art Understanding](https://mlsanity.com/paper/2505.06020/) (2025)
- [VQArt-Bench: A semantically rich VQA Benchmark for Art and Cultural Heritage](https://mlsanity.com/paper/2510.12750/) (2025)
- [A Challenging Benchmark of Anime Style Recognition](https://mlsanity.com/paper/2204.14034/) (2022)
- [Semi-supervised Human Pose Estimation in Art-historical Images](https://mlsanity.com/paper/2207.02976/) (2022)
- [CL-CrossVQA: A Continual Learning Benchmark for Cross-Domain Visual Question Answering](https://mlsanity.com/paper/2211.10567/) (2022)
- [Poses of People in Art: A Data Set for Human Pose Estimation in Digital Art History](https://mlsanity.com/paper/2301.05124/) (2023)
- [SniffyArt: The Dataset of Smelling Persons](https://mlsanity.com/paper/2311.11888/) (2023)
- [Multimodal Metadata Assignment for Cultural Heritage Artifacts](https://mlsanity.com/paper/2406.00423/) (2024)

## Abstract (arXiv)

Automatic art analysis has been mostly focused on classifying artworks into different artistic styles. However, understanding an artistic representation involves more complex processes, such as identifying the elements in the scene or recognizing author influences. We present SemArt, a multi-modal dataset for semantic art understanding. SemArt is a collection of fine-art painting images in which each image is associated to a number of attributes and a textual artistic comment, such as those that appear in art catalogues or museum collections. To evaluate semantic art understanding, we envisage the Text2Art challenge, a multi-modal retrieval task where relevant paintings are retrieved according to an artistic text, and vice versa. We also propose several models for encoding visual and textual artistic representations into a common semantic space. Our best approach is able to find the correct image within the top 10 ranked images in the 45.5% of the test samples. Moreover, our models show remarkable levels of art understanding when compared against human evaluation.