Does CLIP perceive art the same way we do?
Not on the map. Why?
Abstract (arXiv)
CLIP has emerged as a powerful multimodal model capable of connecting images and text through joint embeddings, but to what extent does it 'see' the same way humans do - especially when interpreting artworks? In this paper, we investigate CLIP's ability to extract high-level semantic and stylistic information from paintings, including both human-created and AI-generated imagery. We evaluate its perception across multiple dimensions: content, scene understanding, artistic style, historical period, and the presence of visual deformations or artifacts. By designing targeted probing tasks and comparing CLIP's responses to human annotations and expert benchmarks, we explore its alignment with human perceptual and contextual understanding. Our findings reveal both strengths and limitations in CLIP's visual representations, particularly in relation to aesthetic cues and artistic intent. We further discuss the implications of these insights for using CLIP as a guidance mechanism during generative processes, such as style transfer or prompt-based image synthesis. Our work highlights the need for deeper interpretability in multimodal systems, especially when applied to creative domains where nuance and subjectivity play a central role.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 8 May 2025.
Builds on
31 of 42 references.
References · 31
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
- 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
- 2021Learning to Prompt for Vision-Language Modelscited by 2,281
- 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
- 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
- 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
- 2014Understanding Deep Image Representations by Inverting Themcited by 403
- 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidancecited by 212
19 more
- 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 187
- 2022CLIP-ReID: Exploiting Vision-Language Model for Image Re-Identification without Concrete Text Labelscited by 181
- 2015Inverting Visual Representations with Convolutional Networkscited by 170
- 2022Task Residual for Tuning Vision-Language Modelscited by 123
- 2023Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinementcited by 84
- 2023Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Featurescited by 52
- 2024ClipSAM: CLIP and SAM Collaboration for Zero-Shot Anomaly Segmentationcited by 49
- 2023A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidancecited by 40
- 2022Contrastive Language-Image Pre-Training with Knowledge Graphscited by 34
- 2022SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classificationcited by 34
- 2024Selective Vision-Language Subspace Projection for Few-shot CLIPcited by 18
- 2024CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decodingcited by 13
- 2024What do we learn from inverting CLIP models?cited by 13
- 2024Toward a Holistic Evaluation of Robustness in CLIP Modelscited by 13
- 2024Exploring the Adversarial Robustness of CLIP for AI-generated Image Detectioncited by 11
- 2025A Critical Assessment of Modern Generative Models' Ability to Replicate Artistic Stylescited by 6
- 2024Faster Image2Video Generation: A Closer Look at CLIP Image Embedding's Impact on Spatio-Temporal Cross-Attentionscited by 4
- 2025Analyzing CLIP's Performance Limitations in Multi-Object Scenarios: A Controlled High-Resolution Studycited by 4
- 2024Context-Aware Multimodal Pretrainingcited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.