# Semantic Segmentation in Art Paintings

arXiv: https://arxiv.org/abs/2203.03238 · HTML: https://mlsanity.com/paper/2203.03238/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 74 references that are in the map.
- First posted on arXiv 7 Mar 2022.

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning](https://mlsanity.com/paper/1805.04687/) (2018)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Learning to Adapt Structured Output Space for Semantic Segmentation](https://mlsanity.com/paper/1802.10349/) (2018)
- [Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation](https://mlsanity.com/paper/1909.11065/) (2019)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [A Neural Algorithm of Artistic Style](https://mlsanity.com/paper/1508.06576/) (2015)
- [Bidirectional Learning for Domain Adaptation of Semantic Segmentation](https://mlsanity.com/paper/1904.10620/) (2019)
- [High-Resolution Representations for Labeling Pixels and Regions](https://mlsanity.com/paper/1904.04514/) (2019)
- [Classes Matter: A Fine-grained Adversarial Approach to Cross-domain Semantic Segmentation](https://mlsanity.com/paper/2007.09222/) (2020)
- [Gradually Vanishing Bridge for Adversarial Domain Adaptation](https://mlsanity.com/paper/2003.13183/) (2020)
- [No More Discrimination: Cross City Adaptation of Road Scene Segmenters](https://mlsanity.com/paper/1704.08509/) (2017)
- [Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis](https://mlsanity.com/paper/1601.04589/) (2016)
- [Open Compound Domain Adaptation](https://mlsanity.com/paper/1909.03403/) (2019)
- [FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation](https://mlsanity.com/paper/1903.11816/) (2019)

## Built on by (most cited first, 23 of 23)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2303.11797/) (2023)
- [DICEPTION: A Generalist Diffusion Model for Visual Perceptual Tasks](https://mlsanity.com/paper/2502.17157/) (2025)
- [What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2306.15521/) (2023)
- [ZIM: Zero-Shot Image Matting for Anything](https://mlsanity.com/paper/2411.00626/) (2024)
- [SAM2 for Image and Video Segmentation: A Comprehensive Survey](https://mlsanity.com/paper/2503.12781/) (2025)
- [Stable Segment Anything Model](https://mlsanity.com/paper/2311.15776/) (2023)
- [High-Quality Mask Tuning Matters for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2412.11464/) (2024)
- [Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation](https://mlsanity.com/paper/2503.19647/) (2025)
- [MoPEFT: A Mixture-of-PEFTs for the Segment Anything Model](https://mlsanity.com/paper/2405.00293/) (2024)
- [Test-Time Optimization for Domain Adaptive Open Vocabulary Segmentation](https://mlsanity.com/paper/2501.04696/) (2025)
- [Controllable Multi-domain Semantic Artwork Synthesis](https://mlsanity.com/paper/2308.10111/) (2023)
- [gen2seg: Generative Models Enable Generalizable Instance Segmentation](https://mlsanity.com/paper/2505.15263/) (2025)
- [Artificial Intelligence for Geometry-Based Feature Extraction, Analysis and Synthesis in Artistic Images: A Survey](https://mlsanity.com/paper/2412.01450/) (2024)
- [M2N2V2: Multi-Modal Unsupervised and Training-free Interactive Segmentation](https://mlsanity.com/paper/2503.16254/) (2025)
- [Are Image Generators Zero-Shot Perceivers? A Rigorous Evaluation](https://mlsanity.com/paper/2609.07884/) (2026)
- [ARTeFACT: Benchmarking Segmentation Models on Diverse Analogue Media Damage](https://mlsanity.com/paper/2412.04580/) (2024)
- [Segment Anything with Robust Uncertainty-Accuracy Correlation](https://mlsanity.com/paper/2605.10603/) (2026)
- [Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Tasks](https://mlsanity.com/paper/2606.10967/) (2026)
- [Prompting Diffusion Models for Zero-Shot Instance Segmentation](https://mlsanity.com/paper/2606.22660/) (2026)
- [Learning visual representations for compositional analysis of artworks and photographs](https://mlsanity.com/paper/2608.06142/) (2026)
- [Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement](https://mlsanity.com/paper/2609.34528/) (2026)

## Abstract (arXiv)

Semantic segmentation is a difficult task even when trained in a supervised manner on photographs. In this paper, we tackle the problem of semantic segmentation of artistic paintings, an even more challenging task because of a much larger diversity in colors, textures, and shapes and because there are no ground truth annotations available for segmentation. We propose an unsupervised method for semantic segmentation of paintings using domain adaptation. Our approach creates a training set of pseudo-paintings in specific artistic styles by using style-transfer on the PASCAL VOC 2012 dataset, and then applies domain confusion between PASCAL VOC 2012 and real paintings. These two steps build on a new dataset we gathered called DRAM (Diverse Realism in Art Movements) composed of figurative art paintings from four movements, which are highly diverse in pattern, color, and geometry. To segment new paintings, we present a composite multi-domain adaptation method that trains on each sub-domain separately and composes their solutions during inference time. Our method provides better segmentation results not only on the specific artistic movements of DRAM, but also on other, unseen ones. We compare our approach to alternative methods and show applications of semantic segmentation in art paintings. The code and models for our approach are publicly available at: https://github.com/Nadavc220/SemanticSegmentationInArtPaintings.