# ArtRAG: Retrieval-Augmented Generation with Structured Context for Visual Art Understanding

arXiv: https://arxiv.org/abs/2505.06020 · HTML: https://mlsanity.com/paper/2505.06020/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 25 of its 61 references that are in the map.
- First posted on arXiv 9 May 2025.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Generative Multimodal Models are In-Context Learners](https://mlsanity.com/paper/2312.13286/) (2023)
- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [Recommender Systems in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2307.02046/) (2023)
- [ArtEmis: Affective Language for Visual Art](https://mlsanity.com/paper/2101.07396/) (2021)
- [Retrieving Multimodal Information for Augmented Generation: A Survey](https://mlsanity.com/paper/2303.10868/) (2023)
- [How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval](https://mlsanity.com/paper/1810.09617/) (2018)
- [CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification](https://mlsanity.com/paper/2204.14244/) (2022)
- [A Dataset and Baselines for Visual Question Answering on Art](https://mlsanity.com/paper/2008.12520/) (2020)
- [It is Okay to Not Be Okay: Overcoming Emotional Bias in Affective Image Captioning by Contrastive Data Collection](https://mlsanity.com/paper/2204.07660/) (2022)
- [Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation](https://mlsanity.com/paper/2109.05743/) (2021)
- [GalleryGPT: Analyzing Paintings with Large Multimodal Models](https://mlsanity.com/paper/2408.00491/) (2024)
- [Paint4Poem: A Dataset for Artistic Visualization of Classical Chinese Poems](https://mlsanity.com/paper/2109.11682/) (2021)
- [Iconographic Image Captioning for Artworks](https://mlsanity.com/paper/2102.03942/) (2021)
- [ArtGPT-4: Towards Artistic-understanding Large Vision-Language Models with Enhanced Adapter](https://mlsanity.com/paper/2305.07490/) (2023)
- [Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2411.05706/) (2024)
- [Graph Neural Networks for Knowledge Enhanced Visual Representation of Paintings](https://mlsanity.com/paper/2105.08190/) (2021)
- [Im2Oil: Stroke-Based Oil Painting Rendering with Linearly Controllable Fineness Via Adaptive Sampling](https://mlsanity.com/paper/2209.13219/) (2022)
- [Towards Artwork Explanation in Large-scale Vision Language Models](https://mlsanity.com/paper/2403.00068/) (2024)

## Built on by (most cited first, 7 of 7)

- [ViTCoT: Video-Text Interleaved Chain-of-Thought for Boosting Video Understanding in Large Language Models](https://mlsanity.com/paper/2507.09876/) (2025)
- [Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understanding](https://mlsanity.com/paper/2508.20765/) (2025)
- [VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings](https://mlsanity.com/paper/2603.02435/) (2026)
- [A-MAR: Agent-based Multimodal Art Retrieval for Fine-Grained Artwork Understanding](https://mlsanity.com/paper/2604.19689/) (2026)
- [On Semiotic-Grounded Interpretive Evaluation of Generative Art](https://mlsanity.com/paper/2604.08641/) (2026)
- [MMArt: A Multi-Perspective Multimodal Dataset for Visual Art Understanding](https://mlsanity.com/paper/2608.10706/) (2026)
- [Sanyu Studio: A Multi-Agent System for Art-Historical Narrative Construction](https://mlsanity.com/paper/2608.18677/) (2026)

## Abstract (arXiv)

Understanding visual art requires reasoning across multiple perspectives -- cultural, historical, and stylistic -- beyond mere object recognition. While recent multimodal large language models (MLLMs) perform well on general image captioning, they often fail to capture the nuanced interpretations that fine art demands. We propose ArtRAG, a novel, training-free framework that combines structured knowledge with retrieval-augmented generation (RAG) for multi-perspective artwork explanation. ArtRAG automatically constructs an Art Context Knowledge Graph (ACKG) from domain-specific textual sources, organizing entities such as artists, movements, themes, and historical events into a rich, interpretable graph. At inference time, a multi-granular structured retriever selects semantically and topologically relevant subgraphs to guide generation. This enables MLLMs to produce contextually grounded, culturally informed art descriptions. Experiments on the SemArt and Artpedia datasets show that ArtRAG outperforms several heavily trained baselines. Human evaluations further confirm that ArtRAG generates coherent, insightful, and culturally enriched interpretations.