# Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation

arXiv: https://arxiv.org/abs/2109.05743 · HTML: https://mlsanity.com/paper/2109.05743/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 78 references that are in the map.
- First posted on arXiv 13 Sep 2021.

## Builds on (most cited first, 30 of 38)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [Skip-Thought Vectors](https://mlsanity.com/paper/1506.06726/) (2015)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [Neural Baby Talk](https://mlsanity.com/paper/1803.09845/) (2018)
- [How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval](https://mlsanity.com/paper/1810.09617/) (2018)
- [Good News, Everyone! Context driven entity-aware captioning for news images](https://mlsanity.com/paper/1904.01475/) (2019)
- [Recognizing Image Style](https://mlsanity.com/paper/1311.3715/) (2013)
- [Towards Diverse and Natural Image Descriptions via a Conditional GAN](https://mlsanity.com/paper/1703.06029/) (2017)
- [Inverse Cooking: Recipe Generation from Food Images](https://mlsanity.com/paper/1812.06164/) (2018)
- [Image Captioning and Visual Question Answering Based on Attributes and External Knowledge](https://mlsanity.com/paper/1603.02814/) (2016)
- [BAM! The Behance Artistic Media Dataset for Recognition Beyond Photography](https://mlsanity.com/paper/1704.08614/) (2017)
- [A Dataset and Baselines for Visual Question Answering on Art](https://mlsanity.com/paper/2008.12520/) (2020)
- [Weakly Supervised Object Detection in Artworks](https://mlsanity.com/paper/1810.02569/) (2018)

## Built on by (most cited first, 30 of 30)

- [Measuring Style Similarity in Diffusion Models](https://mlsanity.com/paper/2404.01292/) (2024)
- [One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos](https://mlsanity.com/paper/2409.19603/) (2024)
- [SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text](https://mlsanity.com/paper/2204.11964/) (2022)
- [Not Only Generative Art: Stable Diffusion for Content-Style Disentanglement in Art Analysis](https://mlsanity.com/paper/2304.10278/) (2023)
- [Diffusion-Based Visual Art Creation: A Survey and New Perspectives](https://mlsanity.com/paper/2408.12128/) (2024)
- [Impossible Videos](https://mlsanity.com/paper/2503.14378/) (2025)
- [StyleBabel: Artistic Style Tagging and Captioning](https://mlsanity.com/paper/2203.05321/) (2022)
- [Towards Artwork Explanation in Large-scale Vision Language Models](https://mlsanity.com/paper/2403.00068/) (2024)
- [ArtRAG: Retrieval-Augmented Generation with Structured Context for Visual Art Understanding](https://mlsanity.com/paper/2505.06020/) (2025)
- [Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach](https://mlsanity.com/paper/2408.07249/) (2024)
- [ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval](https://mlsanity.com/paper/2507.21917/) (2025)
- [Training A Small Emotional Vision Language Model for Visual Art Comprehension](https://mlsanity.com/paper/2403.11150/) (2024)
- [KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph](https://mlsanity.com/paper/2409.10921/) (2024)
- [How Well Do Vision Transformers (VTs) Transfer To The Non-Natural Image Domain? An Empirical Study Involving Art Classification](https://mlsanity.com/paper/2208.04693/) (2022)
- [Generating image captions with external encyclopedic knowledge](https://mlsanity.com/paper/2210.04806/) (2022)
- [Automatic Analysis of Human Body Representations in Western Art](https://mlsanity.com/paper/2210.08860/) (2022)
- [Taming CLIP for Fine-grained and Structured Visual Understanding of Museum Exhibits](https://mlsanity.com/paper/2409.01690/) (2024)
- [No Annotations for Object Detection in Art through Stable Diffusion](https://mlsanity.com/paper/2412.06286/) (2024)
- [CompArt: Operationalizing Aesthetic Alignment in Text-to-Image Generation via Principles of Art](https://mlsanity.com/paper/2503.12018/) (2025)
- [VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings](https://mlsanity.com/paper/2603.02435/) (2026)
- [Artificial Intelligence for Geometry-Based Feature Extraction, Analysis and Synthesis in Artistic Images: A Survey](https://mlsanity.com/paper/2412.01450/) (2024)
- [TextTIGER: Text-based Intelligent Generation with Entity Prompt Refinement for Text-to-Image Generation](https://mlsanity.com/paper/2504.18269/) (2025)
- [HanMoVLM: Large Vision-Language Models for Professional Artistic Painting Evaluation](https://mlsanity.com/paper/2603.10814/) (2026)
- [A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding](https://mlsanity.com/paper/2604.19689/) (2026)
- [Art Beyond Semantics: Sheaf-Informed Contrastive Learning for Multi-Relational Representations](https://mlsanity.com/paper/2607.16321/) (2026)
- [Set2Seq Transformer: Temporal and Position-Aware Set Representations for Sequential Multiple-Instance Learning](https://mlsanity.com/paper/2408.03404/) (2024)
- [ImageSet2Text: Describing Sets of Images through Text](https://mlsanity.com/paper/2503.19361/) (2025)
- [Art2Mus: Artwork-to-Music Generation via Visual Conditioning and Large-Scale Cross-Modal Alignment](https://mlsanity.com/paper/2602.17599/) (2026)
- [Learning visual representations for compositional analysis of artworks and photographs](https://mlsanity.com/paper/2608.06142/) (2026)
- [MMArt: A Multi-Perspective Multimodal Dataset for Visual Art Understanding](https://mlsanity.com/paper/2608.10706/) (2026)

## Abstract (arXiv)

Have you ever looked at a painting and wondered what is the story behind it? This work presents a framework to bring art closer to people by generating comprehensive descriptions of fine-art paintings. Generating informative descriptions for artworks, however, is extremely challenging, as it requires to 1) describe multiple aspects of the image such as its style, content, or composition, and 2) provide background and contextual knowledge about the artist, their influences, or the historical period. To address these challenges, we introduce a multi-topic and knowledgeable art description framework, which modules the generated sentences according to three artistic topics and, additionally, enhances each description with external knowledge. The framework is validated through an exhaustive analysis, both quantitative and qualitative, as well as a comparative human evaluation, demonstrating outstanding results in terms of both topic diversity and information veracity.