# A Dataset and Baselines for Visual Question Answering on Art

arXiv: https://arxiv.org/abs/2008.12520 · HTML: https://mlsanity.com/paper/2008.12520/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2020.

## Built on by (most cited first, 30 of 38)

- [QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension](https://mlsanity.com/paper/2107.12708/) (2021)
- [Demystifying Chains, Trees, and Graphs of Thoughts](https://mlsanity.com/paper/2401.14295/) (2024)
- [A Systematic Survey of Automatic Prompt Optimization Techniques](https://mlsanity.com/paper/2502.16923/) (2025)
- [Understanding and Creating Art with AI: Review and Outlook](https://mlsanity.com/paper/2102.09109/) (2021)
- [FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning](https://mlsanity.com/paper/2308.12305/) (2023)
- [Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation](https://mlsanity.com/paper/2109.05743/) (2021)
- [Data Contamination Calibration for Black-box LLMs](https://mlsanity.com/paper/2405.11930/) (2024)
- [Modern Question Answering Datasets and Benchmarks: A Survey](https://mlsanity.com/paper/2206.15030/) (2022)
- [Q: How to Specialize Large Vision-Language Models to Data-Scarce VQA Tasks? A: Self-Train on Unlabeled Images!](https://mlsanity.com/paper/2306.03932/) (2023)
- [Iconographic Image Captioning for Artworks](https://mlsanity.com/paper/2102.03942/) (2021)
- [Diffusion Based Augmentation for Captioning and Retrieval in Cultural Heritage](https://mlsanity.com/paper/2308.07151/) (2023)
- [Exploring Question Decomposition for Zero-Shot VQA](https://mlsanity.com/paper/2310.17050/) (2023)
- [A survey on knowledge-enhanced multimodal learning](https://mlsanity.com/paper/2211.12328/) (2022)
- [Unleashing the Potential of Large Language Model: Zero-shot VQA for Flood Disaster Scenario](https://mlsanity.com/paper/2312.01882/) (2023)
- [Towards Artwork Explanation in Large-scale Vision Language Models](https://mlsanity.com/paper/2403.00068/) (2024)
- [ArtRAG: Retrieval-Augmented Generation with Structured Context for Visual Art Understanding](https://mlsanity.com/paper/2505.06020/) (2025)
- [Is GPT-3 all you need for Visual Question Answering in Cultural Heritage?](https://mlsanity.com/paper/2207.12101/) (2022)
- [CL-CrossVQA: A Continual Learning Benchmark for Cross-Domain Visual Question Answering](https://mlsanity.com/paper/2211.10567/) (2022)
- [Poses of People in Art: A Data Set for Human Pose Estimation in Digital Art History](https://mlsanity.com/paper/2301.05124/) (2023)
- [Weakly Supervised Attended Object Detection Using Gaze Data as Annotations](https://mlsanity.com/paper/2204.07090/) (2022)
- [ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval](https://mlsanity.com/paper/2507.21917/) (2025)
- [IICONGRAPH: improved Iconographic and Iconological Statements in Knowledge Graphs](https://mlsanity.com/paper/2402.00048/) (2024)
- [Training A Small Emotional Vision Language Model for Visual Art Comprehension](https://mlsanity.com/paper/2403.11150/) (2024)
- [VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes](https://mlsanity.com/paper/2509.25339/) (2025)
- [Impressions: Understanding Visual Semiotics and Aesthetic Impact](https://mlsanity.com/paper/2310.17887/) (2023)
- [Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style](https://mlsanity.com/paper/2603.11024/) (2026)
- [Taming CLIP for Fine-grained and Structured Visual Understanding of Museum Exhibits](https://mlsanity.com/paper/2409.01690/) (2024)
- [No Annotations for Object Detection in Art through Stable Diffusion](https://mlsanity.com/paper/2412.06286/) (2024)
- [Smelly, dense, and spreaded: The Object Detection for Olfactory References (ODOR) dataset](https://mlsanity.com/paper/2507.08384/) (2025)
- [MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosis](https://mlsanity.com/paper/2508.08275/) (2025)

## Abstract (arXiv)

Answering questions related to art pieces (paintings) is a difficult task, as it implies the understanding of not only the visual information that is shown in the picture, but also the contextual knowledge that is acquired through the study of the history of art. In this work, we introduce our first attempt towards building a new dataset, coined AQUA (Art QUestion Answering). The question-answer (QA) pairs are automatically generated using state-of-the-art question generation methods based on paintings and comments provided in an existing art understanding dataset. The QA pairs are cleansed by crowdsourcing workers with respect to their grammatical correctness, answerability, and answers' correctness. Our dataset inherently consists of visual (painting-based) and knowledge (comment-based) questions. We also present a two-branch model as baseline, where the visual and knowledge questions are handled independently. We extensively compare our baseline model against the state-of-the-art models for question answering, and we provide a comprehensive study about the challenges and potential future directions for visual question answering on art.