# Context-Infused Visual Grounding for Art

arXiv: https://arxiv.org/abs/2410.12369 · HTML: https://mlsanity.com/paper/2410.12369/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 65 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [Cross-Domain Weakly-Supervised Object Detection through Progressive Domain Adaptation](https://mlsanity.com/paper/1803.11365/) (2018)
- [Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature](https://mlsanity.com/paper/1505.00855/) (2015)
- [DetCLIP: Dictionary-Enriched Visual-Concept Paralleled Pre-training for Open-world Detection](https://mlsanity.com/paper/2209.09407/) (2022)
- [Learning Deep Structure-Preserving Image-Text Embeddings](https://mlsanity.com/paper/1511.06078/) (2015)
- [Linguistic Structure Guided Context Modeling for Referring Image Segmentation](https://mlsanity.com/paper/2010.00515/) (2020)
- [How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval](https://mlsanity.com/paper/1810.09617/) (2018)
- [CLEVR-Ref+: Diagnosing Visual Reasoning with Referring Expressions](https://mlsanity.com/paper/1901.00850/) (2019)
- [Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoning](https://mlsanity.com/paper/2205.00272/) (2022)
- [Graph-Structured Referring Expression Reasoning in The Wild](https://mlsanity.com/paper/2004.08814/) (2020)
- [Weakly Supervised Object Detection in Artworks](https://mlsanity.com/paper/1810.02569/) (2018)
- [Detecting People in Artwork with CNNs](https://mlsanity.com/paper/1610.08871/) (2016)
- [KaoKore: A Pre-modern Japanese Art Facial Expression Dataset](https://mlsanity.com/paper/2002.08595/) (2020)
- [A Data Set and a Convolutional Model for Iconography Classification in Paintings](https://mlsanity.com/paper/2010.11697/) (2020)
- [Advancing Visual Grounding with Scene Knowledge: Benchmark and Method](https://mlsanity.com/paper/2307.11558/) (2023)
- [OmniArt: Multi-task Deep Learning for Artistic Data Analysis](https://mlsanity.com/paper/1708.00684/) (2017)
- [Context-Aware Embeddings for Automatic Art Analysis](https://mlsanity.com/paper/1904.04985/) (2019)
- [Iconographic Image Captioning for Artworks](https://mlsanity.com/paper/2102.03942/) (2021)
- [Improving Object Detection in Art Images Using Only Style Transfer](https://mlsanity.com/paper/2102.06529/) (2021)
- [Multiple instance learning on deep features for weakly supervised object detection with extreme domain shifts](https://mlsanity.com/paper/2008.01178/) (2020)
- [Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks](https://mlsanity.com/paper/1704.00260/) (2017)

## Built on by (most cited first, 2 of 2)

- [No Annotations for Object Detection in Art through Stable Diffusion](https://mlsanity.com/paper/2412.06286/) (2024)
- ["Jutters"](https://mlsanity.com/paper/2601.11532/) (2025)

## Abstract (arXiv)

Many artwork collections contain textual attributes that provide rich and contextualised descriptions of artworks. Visual grounding offers the potential for localising subjects within these descriptions on images, however, existing approaches are trained on natural images and generalise poorly to art. In this paper, we present CIGAr (Context-Infused GroundingDINO for Art), a visual grounding approach which utilises the artwork descriptions during training as context, thereby enabling visual grounding on art. In addition, we present a new dataset, Ukiyo-eVG, with manually annotated phrase-grounding annotations, and we set a new state-of-the-art for object detection on two artwork datasets.