# VQArt-Bench: A semantically rich VQA Benchmark for Art and Cultural Heritage

arXiv: https://arxiv.org/abs/2510.12750 · HTML: https://mlsanity.com/paper/2510.12750/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 29 of its 42 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 29 of 29)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [Kimi-VL Technical Report](https://mlsanity.com/paper/2504.07491/) (2025)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [Woodpecker: Hallucination Correction for Multimodal Large Language Models](https://mlsanity.com/paper/2310.16045/) (2023)
- [Pixtral 12B](https://mlsanity.com/paper/2410.07073/) (2024)
- [Exploring Models and Data for Image Question Answering](https://mlsanity.com/paper/1505.02074/) (2015)
- [Aria: An Open Multimodal Native Mixture-of-Experts Model](https://mlsanity.com/paper/2410.05993/) (2024)
- [Visual Question Answering: A Survey of Methods and Datasets](https://mlsanity.com/paper/1607.05910/) (2016)
- [How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval](https://mlsanity.com/paper/1810.09617/) (2018)
- [Aya Vision: Advancing the Frontier of Multilingual Multimodality](https://mlsanity.com/paper/2505.08751/) (2025)
- [Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation](https://mlsanity.com/paper/2006.07585/) (2020)
- [Exploring Long Tail Visual Relationship Recognition with Large Vocabulary](https://mlsanity.com/paper/2004.00436/) (2020)
- [Learning Answer Embeddings for Visual Question Answering](https://mlsanity.com/paper/1806.03724/) (2018)
- [Learning Visual Knowledge Memory Networks for Visual Question Answering](https://mlsanity.com/paper/1806.04860/) (2018)
- [Visual Question Answering for Cultural Heritage](https://mlsanity.com/paper/2003.09853/) (2020)
- [The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering](https://mlsanity.com/paper/2501.07109/) (2025)

## Built on by (most cited first, 7 of 7)

- [PPTArena: A Benchmark for PowerPoint Editing](https://mlsanity.com/paper/2512.03042/) (2025)
- [UNICBench: UNIfied Counting Benchmark for MLLM](https://mlsanity.com/paper/2603.00595/) (2026)
- [HanMoVLM: Large Vision-Language Models for Professional Artistic Painting Evaluation](https://mlsanity.com/paper/2603.10814/) (2026)
- [Understanding How MLLMs Describe Artworks Using Token Activation Maps](https://mlsanity.com/paper/2606.27947/) (2026)
- [EduArt: An educational-level benchmark for evaluating art history knowledge in large language models](https://mlsanity.com/paper/2607.02007/) (2026)
- [VaseMuseum: Digital Intelligent Museum for Ancient Greek Pottery](https://mlsanity.com/paper/2607.06374/) (2026)
- [MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education](https://mlsanity.com/paper/2609.19088/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have demonstrated significant capabilities in joint visual and linguistic tasks. However, existing Visual Question Answering (VQA) benchmarks often fail to evaluate deep semantic understanding, particularly in complex domains like visual art analysis. Confined to simple syntactic structures and surface-level attributes, these questions fail to capture the diversity and depth of human visual inquiry. This limitation incentivizes models to exploit statistical shortcuts rather than engage in visual reasoning. To address this gap, we introduce VQArt-Bench, a new, large-scale VQA benchmark for the cultural heritage domain. This benchmark is constructed using a novel multi-agent pipeline where specialized agents collaborate to generate nuanced, validated, and linguistically diverse questions. The resulting benchmark is structured along relevant visual understanding dimensions that probe a model's ability to interpret symbolic meaning, narratives, and complex visual relationships. Our evaluation of 14 state-of-the-art MLLMs on this benchmark reveals significant limitations in current models, including a surprising weakness in simple counting tasks and a clear performance gap between proprietary and open-source models.