# GalleryGPT: Analyzing Paintings with Large Multimodal Models

arXiv: https://arxiv.org/abs/2408.00491 · HTML: https://mlsanity.com/paper/2408.00491/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 16 of its 58 references that are in the map.
- First posted on arXiv 1 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 16 of 16)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2405.06211/) (2024)
- [Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models](https://mlsanity.com/paper/2406.17294/) (2024)
- [The ArtBench Dataset: Benchmarking Generative Models with Artworks](https://mlsanity.com/paper/2206.11404/) (2022)
- [Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives](https://mlsanity.com/paper/2406.05615/) (2024)

## Built on by (most cited first, 28 of 28)

- [Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs](https://mlsanity.com/paper/2504.17432/) (2025)
- [STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding](https://mlsanity.com/paper/2503.15973/) (2025)
- [Have Large Vision-Language Models Mastered Art History?](https://mlsanity.com/paper/2409.03521/) (2024)
- [ArtRAG: Retrieval-Augmented Generation with Structured Context for Visual Art Understanding](https://mlsanity.com/paper/2505.06020/) (2025)
- [Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images](https://mlsanity.com/paper/2502.16593/) (2025)
- [ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrieval](https://mlsanity.com/paper/2507.21917/) (2025)
- [ChartAdapter: Large Vision-Language Model for Chart Summarization](https://mlsanity.com/paper/2412.20715/) (2024)
- [Multimodal Mathematical Reasoning with Diverse Solving Perspective](https://mlsanity.com/paper/2507.02804/) (2025)
- [A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Tasks](https://mlsanity.com/paper/2509.23208/) (2025)
- [Assessing LLMs in Art Contexts: Critique Generation and Theory of Mind Evaluation](https://mlsanity.com/paper/2504.12805/) (2025)
- [WP-CLIP: Leveraging CLIP to Predict W"olfflin's Principles in Visual Art](https://mlsanity.com/paper/2508.12668/) (2025)
- [More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration](https://mlsanity.com/paper/2510.02227/) (2025)
- [StyleDecoupler: Generalizable Artistic Style Disentanglement](https://mlsanity.com/paper/2601.17697/) (2026)
- [SIF: Semantically In-Distribution Fingerprints for Large Vision-Language Models](https://mlsanity.com/paper/2604.17041/) (2026)
- [Identifying Prompted Artist Names from Generated Images](https://mlsanity.com/paper/2507.18633/) (2025)
- [Physical Plausibility Reasoning via HCM-GRPO: Empowering Compact Model for Superior Performance](https://mlsanity.com/paper/2511.10055/) (2025)
- [Echoes of ownership: Adversarial-guided dual injection for copyright protection in MLLMs](https://mlsanity.com/paper/2602.18845/) (2026)
- [HanMoVLM: Large Vision-Language Models for Professional Artistic Painting Evaluation](https://mlsanity.com/paper/2603.10814/) (2026)
- [A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding](https://mlsanity.com/paper/2604.19689/) (2026)
- [Beyond Starry Night: Shortcut-Aware Control-State Planning for Artist-Grounded Text to Image Generation](https://mlsanity.com/paper/2608.06751/) (2026)
- [Set2Seq Transformer: Temporal and Position-Aware Set Representations for Sequential Multiple-Instance Learning](https://mlsanity.com/paper/2408.03404/) (2024)
- [Speaking images. A novel framework for the automated self-description of artworks](https://mlsanity.com/paper/2506.05368/) (2025)
- [ArchGPT: Understanding the World's Architectures with Large Multimodal Models](https://mlsanity.com/paper/2509.20858/) (2025)
- [Data-Efficient Brushstroke Generation with Diffusion Models for Oil Painting](https://mlsanity.com/paper/2603.01103/) (2026)
- [M-ArtAgent: Evidence-Based Multimodal Agent for Implicit Art Influence Discovery](https://mlsanity.com/paper/2604.07468/) (2026)
- [LLMs Should Incorporate Explicit Mechanisms for Human Empathy](https://mlsanity.com/paper/2604.10557/) (2026)
- [Understanding How MLLMs Describe Artworks Using Token Activation Maps](https://mlsanity.com/paper/2606.27947/) (2026)
- [MMArt: A Multi-Perspective Multimodal Dataset for Visual Art Understanding](https://mlsanity.com/paper/2608.10706/) (2026)

## Abstract (arXiv)

Artwork analysis is important and fundamental skill for art appreciation, which could enrich personal aesthetic sensibility and facilitate the critical thinking ability. Understanding artworks is challenging due to its subjective nature, diverse interpretations, and complex visual elements, requiring expertise in art history, cultural background, and aesthetic theory. However, limited by the data collection and model ability, previous works for automatically analyzing artworks mainly focus on classification, retrieval, and other simple tasks, which is far from the goal of AI. To facilitate the research progress, in this paper, we step further to compose comprehensive analysis inspired by the remarkable perception and generation ability of large multimodal models. Specifically, we first propose a task of composing paragraph analysis for artworks, i.e., painting in this paper, only focusing on visual characteristics to formulate more comprehensive understanding of artworks. To support the research on formal analysis, we collect a large dataset PaintingForm, with about 19k painting images and 50k analysis paragraphs. We further introduce a superior large multimodal model for painting analysis composing, dubbed GalleryGPT, which is slightly modified and fine-tuned based on LLaVA architecture leveraging our collected data. We conduct formal analysis generation and zero-shot experiments across several datasets to assess the capacity of our model. The results show remarkable performance improvements comparing with powerful baseline LMMs, demonstrating its superb ability of art analysis and generalization. textcolorblueThe codes and model are available at: https://github.com/steven640pixel/GalleryGPT.