Data to 14 Oct 2026

How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval

arXiv 1810.09617 ↗First posted 23 Oct 2018cs.CVcited by 74

Not on the map. Why?

Abstract (arXiv)

Automatic art analysis has been mostly focused on classifying artworks into different artistic styles. However, understanding an artistic representation involves more complex processes, such as identifying the elements in the scene or recognizing author influences. We present SemArt, a multi-modal dataset for semantic art understanding. SemArt is a collection of fine-art painting images in which each image is associated to a number of attributes and a textual artistic comment, such as those that appear in art catalogues or museum collections. To evaluate semantic art understanding, we envisage the Text2Art challenge, a multi-modal retrieval task where relevant paintings are retrieved according to an artistic text, and vice versa. We also propose several models for encoding visual and textual artistic representations into a common semantic space. Our best approach is able to find the correct image within the top 10 ranked images in the 45.5% of the test samples. Moreover, our models show remarkable levels of art understanding when compared against human evaluation.

Key facts. Cited by 74 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 10; 2023: 11; 2024: 7; 2025: 9; 2026: 28.

74 papers.

Sort:

By area:

2024Measuring Style Similarity in Diffusion Modelscited by 109
2021ArtEmis: Affective Language for Visual Artcited by 104
2023Human-Art: A Versatile Human-Centric Dataset Bridging Natural and Artificial Scenescited by 68
2022The ArtBench Dataset: Benchmarking Generative Models with Artworkscited by 64
2021Understanding and Creating Art with AI: Review and Outlookcited by 60
2022The Met Dataset: Instance-level Recognition for Artworkscited by 40
2021Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generationcited by 30
2023LMEye: An Interactive Perception Network for Large Language Modelscited by 27
2023Q: How to Specialize Large Vision-Language Models to Data-Scarce VQA Tasks? A: Self-Train on Unlabeled Images!cited by 19
2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19
2023Not Only Generative Art: Stable Diffusion for Content-Style Disentanglement in Art Analysiscited by 18
2024Conditional LoRA Parameter Generationcited by 16

38 more

2021Iconographic Image Captioning for Artworkscited by 15
2026OpenMAG: A Comprehensive Benchmark for Multimodal-Attributed Graphcited by 13
2022StyleBabel: Artistic Style Tagging and Captioningcited by 12
2021GCNBoost: Artwork Classification by Label Propagation through a Knowledge Graphcited by 11
2022A survey on knowledge-enhanced multimodal learningcited by 11
2023Simple Disentanglement of Style and Content in Visual Representationscited by 10
2023ARTxAI: Explainable Artificial Intelligence Curates Deep Representation Learning for Artistic Images using Fuzzy Techniquescited by 10
2021Graph Neural Networks for Knowledge Enhanced Visual Representation of Paintingscited by 9
2022DEArt: Dataset of European Artcited by 9
2023Prototype-based Dataset Comparisoncited by 9
2025ArtRAG: Retrieval-Augmented Generation with Structured Context for Visual Art Understandingcited by 7
2025VQArt-Bench: A semantically rich VQA Benchmark for Art and Cultural Heritagecited by 7
2022A Challenging Benchmark of Anime Style Recognitioncited by 6
2022Semi-supervised Human Pose Estimation in Art-historical Imagescited by 6
2022CL-CrossVQA: A Continual Learning Benchmark for Cross-Domain Visual Question Answeringcited by 6
2023Poses of People in Art: A Data Set for Human Pose Estimation in Digital Art Historycited by 6
2023SniffyArt: The Dataset of Smelling Personscited by 6
2024Multimodal Metadata Assignment for Cultural Heritage Artifactscited by 5
2025ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrievalcited by 5
2021A deep learning approach to clustering visual artscited by 4
2023Seeing the Intangible: Survey of Image Classification into High-Level and Abstract Categoriescited by 4
2024KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graphcited by 4
2022Learning More May Not Be Better: Knowledge Transferability in Vision and Language Taskscited by 3
2025Assessing LLMs in Art Contexts: Critique Generation and Theory of Mind Evaluationcited by 3
2025Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understandingcited by 3
2026ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledgecited by 3
2026TMTE: Effective Multimodal Graph Learning with Task-aware Modality and Topology Co-evolutioncited by 3
2026STAGE: Tackling Semantic Drift in Multimodal Federated Graph Learningcited by 3
2024Context-Infused Visual Grounding for Artcited by 2
2025Smelly, dense, and spreaded: The Object Detection for Olfactory References (ODOR) datasetcited by 2
2026CAMPA: Efficient and Aligned Multimodal Graph Learning via Decoupled Propagation and Aggregationcited by 2
2026Learning Transferable Topology Priors for Multi-Agent LLM Collaboration Across Domainscited by 2
2026LION: A Clifford Neural Paradigm for Multimodal-Attributed Graph Learningcited by 2
2021Integrating Contextual Knowledge to Visual Features for Fine Art Classificationcited by 1
2021Quantitative analysis of visual representation of sign elements in COVID-19 contextcited by 1
2023Everyone Can Be Picasso? A Computational Framework into the Myth of Human versus AI Paintingcited by 1
2025PPJudge: Towards Human-Aligned Assessment of Artistic Painting Processcited by 1
2026HanMoVLM: Large Vision-Language Models for Professional Artistic Painting Evaluationcited by 1

50 most cited of 74 All 74 (JSON)