mlsanityData to 7 Oct 2026

GalleryGPT: Analyzing Paintings with Large Multimodal Models

arXiv 2408.00491 ↗First posted 1 Aug 2024cs.CL cs.CV cs.MMcited by 28

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Artwork analysis is important and fundamental skill for art appreciation, which could enrich personal aesthetic sensibility and facilitate the critical thinking ability. Understanding artworks is challenging due to its subjective nature, diverse interpretations, and complex visual elements, requiring expertise in art history, cultural background, and aesthetic theory. However, limited by the data collection and model ability, previous works for automatically analyzing artworks mainly focus on classification, retrieval, and other simple tasks, which is far from the goal of AI. To facilitate the research progress, in this paper, we step further to compose comprehensive analysis inspired by the remarkable perception and generation ability of large multimodal models. Specifically, we first propose a task of composing paragraph analysis for artworks, i.e., painting in this paper, only focusing on visual characteristics to formulate more comprehensive understanding of artworks. To support the research on formal analysis, we collect a large dataset PaintingForm, with about 19k painting images and 50k analysis paragraphs. We further introduce a superior large multimodal model for painting analysis composing, dubbed GalleryGPT, which is slightly modified and fine-tuned based on LLaVA architecture leveraging our collected data. We conduct formal analysis generation and zero-shot experiments across several datasets to assess the capacity of our model. The results show remarkable performance improvements comparing with powerful baseline LMMs, demonstrating its superb ability of art analysis and generalization. textcolorblueThe codes and model are available at: https://github.com/steven640pixel/GalleryGPT.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 58 references that are in the map. First posted on arXiv 1 Aug 2024.

Builds on

16 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models25 Jun
  2. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun

Other references · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  8. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  10. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  11. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  12. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
2 more
  1. 2022The ArtBench Dataset: Benchmarking Generative Models with Artworkscited by 64
  2. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 11.

28 papers.

Sort:

By area:

  1. 2025Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMscited by 73
  2. 2025STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understandingcited by 13
  3. 2024Have Large Vision-Language Models Mastered Art History?cited by 9
  4. 2025ArtRAG: Retrieval-Augmented Generation with Structured Context for Visual Art Understandingcited by 7
  5. 2025Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Imagescited by 5
  6. 2025ArtSeek: Deep artwork understanding via multimodal in-context reasoning and late interaction retrievalcited by 5
  7. 2024ChartAdapter: Large Vision-Language Model for Chart Summarizationcited by 4
  8. 2025Multimodal Mathematical Reasoning with Diverse Solving Perspectivecited by 4
  9. 2025A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Taskscited by 4
  10. 2025Assessing LLMs in Art Contexts: Critique Generation and Theory of Mind Evaluationcited by 3
  11. 2025WP-CLIP: Leveraging CLIP to Predict W"olfflin's Principles in Visual Artcited by 3
  12. 2025More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Explorationcited by 3
16 more
  1. 2026StyleDecoupler: Generalizable Artistic Style Disentanglementcited by 3
  2. 2026SIF: Semantically In-Distribution Fingerprints for Large Vision-Language Modelscited by 2
  3. 2025Identifying Prompted Artist Names from Generated Imagescited by 1
  4. 2025Physical Plausibility Reasoning via HCM-GRPO: Empowering Compact Model for Superior Performancecited by 1
  5. 2026Echoes of ownership: Adversarial-guided dual injection for copyright protection in MLLMscited by 1
  6. 2026HanMoVLM: Large Vision-Language Models for Professional Artistic Painting Evaluationcited by 1
  7. 2026A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understandingcited by 1
  8. 2026Beyond Starry Night: Shortcut-Aware Control-State Planning for Artist-Grounded Text to Image Generationcited by 1
  9. 2024Set2Seq Transformer: Temporal and Position-Aware Set Representations for Sequential Multiple-Instance Learningcited by 0
  10. 2025Speaking images. A novel framework for the automated self-description of artworkscited by 0
  11. 2025ArchGPT: Understanding the World's Architectures with Large Multimodal Modelscited by 0
  12. 2026Data-Efficient Brushstroke Generation with Diffusion Models for Oil Paintingcited by 0
  13. 2026M-ArtAgent: Evidence-Based Multimodal Agent for Implicit Art Influence Discoverycited by 0
  14. 2026LLMs Should Incorporate Explicit Mechanisms for Human Empathycited by 0
  15. 2026Understanding How MLLMs Describe Artworks Using Token Activation Mapscited by 0
  16. 2026MMArt: A Multi-Perspective Multimodal Dataset for Visual Art Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CognArtive: Large Language Models for Automating Art Analysis and Decoding Aesthetic Elements
  2. 2023Not Only Generative Art: Stable Diffusion for Content-Style Disentanglement in Art Analysis
  3. 2026Conducting Stylistic Analysis of Paintings through an Art-History Agent
  4. 2023ArtGPT-4: Towards Artistic-understanding Large Vision-Language Models with Enhanced Adapter
  5. 2026Abstract4D: A Large-Scale Dataset and Framework for Understanding the Visual Language of Abstract Art
  6. 2025PPJudge: Towards Human-Aligned Assessment of Artistic Painting Process
  7. 2025Context-aware Multimodal AI Reveals Hidden Pathways in Five Centuries of Art Evolution
  8. 2025The Cow of Rembrandt - Analyzing Artistic Prompt Interpretation in Text-to-Image Models