# Benchmarking Vision-Language and Multimodal Large Language Models in Zero-shot and Few-shot Scenarios: A study on Christian Iconography

arXiv: https://arxiv.org/abs/2509.18839 · HTML: https://mlsanity.com/paper/2509.18839/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 23 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 15 of 15)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [Cross-Domain Weakly-Supervised Object Detection through Progressive Domain Adaptation](https://mlsanity.com/paper/1803.11365/) (2018)
- [Understanding and Creating Art with AI: Review and Outlook](https://mlsanity.com/paper/2102.09109/) (2021)
- [The Met Dataset: Instance-level Recognition for Artworks](https://mlsanity.com/paper/2202.01747/) (2022)
- [Recent progress in semantic image segmentation](https://mlsanity.com/paper/1809.10198/) (2018)
- [Weakly Supervised Object Detection in Artworks](https://mlsanity.com/paper/1810.02569/) (2018)
- [Large Language Models are Good Prompt Learners for Low-Shot Image Classification](https://mlsanity.com/paper/2312.04076/) (2023)
- [ArCo: the Italian Cultural Heritage Knowledge Graph](https://mlsanity.com/paper/1905.02840/) (2019)
- [A Data Set and a Convolutional Model for Iconography Classification in Paintings](https://mlsanity.com/paper/2010.11697/) (2020)
- [DEArt: Dataset of European Art](https://mlsanity.com/paper/2211.01226/) (2022)
- [IICONGRAPH: improved Iconographic and Iconological Statements in Knowledge Graphs](https://mlsanity.com/paper/2402.00048/) (2024)

## Built on by (most cited first, 3 of 3)

- [RB-FT: Rationale-Bootstrapped Fine-Tuning for Video Classification](https://mlsanity.com/paper/2511.15923/) (2025)
- [EduArt: An educational-level benchmark for evaluating art history knowledge in large language models](https://mlsanity.com/paper/2607.02007/) (2026)
- [Large Language Models, Machine-Assisted Cultural Analytics, and the Agentic Turn](https://mlsanity.com/paper/2610.12792/) (2026)

## Abstract (arXiv)

This study evaluates the capabilities of Multimodal Large Language Models (LLMs) and Vision Language Models (VLMs) in the task of single-label classification of Christian Iconography. The goal was to assess whether general-purpose VLMs (CLIP and SigLIP) and LLMs, such as GPT-4o and Gemini 2.5, can interpret the Iconography, typically addressed by supervised classifiers, and evaluate their performance. Two research questions guided the analysis: (RQ1) How do multimodal LLMs perform on image classification of Christian saints? And (RQ2), how does performance vary when enriching input with contextual information or few-shot exemplars? We conducted a benchmarking study using three datasets supporting Iconclass natively: ArtDL, ICONCLASS, and Wikidata, filtered to include the top 10 most frequent classes. Models were tested under three conditions: (1) classification using class labels, (2) classification with Iconclass descriptions, and (3) few-shot learning with five exemplars. Results were compared against ResNet50 baselines fine-tuned on the same datasets. The findings show that Gemini-2.5 Pro and GPT-4o outperformed the ResNet50 baselines. Accuracy dropped significantly on the Wikidata dataset, where Siglip reached the highest accuracy score, suggesting model sensitivity to image size and metadata alignment. Enriching prompts with class descriptions generally improved zero-shot performance, while few-shot learning produced lower results, with only occasional and minimal increments in accuracy. We conclude that general-purpose multimodal LLMs are capable of classification in visually complex cultural heritage domains. These results support the application of LLMs as metadata curation tools in digital humanities workflows, suggesting future research on prompt optimization and the expansion of the study to other classification strategies and models.