On the Explainability of Vision-Language Models in Art History
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Vision-Language Models (VLMs) transfer visual and textual data into a shared embedding space. In so doing, they enable a wide range of multimodal tasks, while also raising critical questions about the nature of machine 'understanding.' In this paper, we examine how Explainable Artificial Intelligence (XAI) methods can render the visual reasoning of a VLM - namely, CLIP - legible in art-historical contexts. To this end, we evaluate seven methods, combining zero-shot localization experiments with human interpretability studies. Our results indicate that, while these methods capture some aspects of human interpretation, their effectiveness hinges on the conceptual stability and representational availability of the examined categories.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 35 references that are in the map. First posted on arXiv 24 Feb 2026.
Builds on
22 of 35 references.
References · 22
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
- 2017Explanation in Artificial Intelligence: Insights from the Social Sciencescited by 1,201
- 2017Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networkscited by 936
- 2019Attention is not Explanationcited by 916
- 2020Quantifying Attention Flow in Transformerscited by 759
- 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 575
- 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
- 2022CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networkscited by 144
- 2021Explainable AI (XAI): A Systematic Meta-Survey of Current Challenges and Future Opportunitiescited by 125
10 more
- 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 116
- 2020Evaluating Weakly Supervised Object Localization Methods Rightcited by 104
- 2023A Closer Look at the Explainability of Contrastive Language-Image Pre-trainingcited by 95
- 2024LeGrad: An Explainability Method for Vision Transformers via Feature Formation Sensitivitycited by 32
- 2018Weakly Supervised Object Detection in Artworkscited by 27
- 2020A Data Set and a Convolutional Model for Iconography Classification in Paintingscited by 17
- 2023CLIP-QDA: An Explainable Concept Bottleneck Modelcited by 13
- 2022DEArt: Dataset of European Artcited by 9
- 2021iART: A Search Engine for Art-Historical Images to Support Research in the Humanitiescited by 6
- 2023Poses of People in Art: A Data Set for Human Pose Estimation in Digital Art Historycited by 6
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Have Large Vision-Language Models Mastered Art History?
- 2026Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Style
- 2024Towards Artwork Explanation in Large-scale Vision Language Models
- 2026On the Cultural Anachronism and Temporal Reasoning in Vision Language Models
- 2026Art Beyond Semantics: Sheaf-Informed Contrastive Learning for Multi-Relational Representations
- 2024Understanding Museum Exhibits using Vision-Language Reasoning
- 2025A Survey on Interpretability in Visual Recognition
- 2026MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education