# CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification

arXiv: https://arxiv.org/abs/2204.14244 · HTML: https://mlsanity.com/paper/2204.14244/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 31 references that are in the map.
- First posted on arXiv 29 Apr 2022.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Lookahead Optimizer: k steps forward, 1 step back](https://mlsanity.com/paper/1907.08610/) (2019)
- [Learning to Navigate for Fine-grained Classification](https://mlsanity.com/paper/1809.00287/) (2018)
- [Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognition](https://mlsanity.com/paper/1903.06150/) (2019)
- [See Better Before Looking Closer: Weakly Supervised Data Augmentation Network for Fine-Grained Visual Classification](https://mlsanity.com/paper/1901.09891/) (2019)
- [Fine-grained Visual-textual Representation Learning](https://mlsanity.com/paper/1709.00340/) (2017)
- [The iMet Collection 2019 Challenge Dataset](https://mlsanity.com/paper/1906.00901/) (2019)
- [Exploring Vision Transformers for Fine-grained Classification](https://mlsanity.com/paper/2106.10587/) (2021)

## Built on by (most cited first, 30 of 65)

- [Swin2SR: SwinV2 Transformer for Compressed Image Super-Resolution and Restoration](https://mlsanity.com/paper/2209.11345/) (2022)
- [Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Features](https://mlsanity.com/paper/2308.11485/) (2023)
- [Multimodal Fake News Detection via CLIP-Guided Learning](https://mlsanity.com/paper/2205.14304/) (2022)
- [CARZero: Cross-Attention Alignment for Radiology Zero-Shot Classification](https://mlsanity.com/paper/2402.17417/) (2024)
- [Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Study](https://mlsanity.com/paper/2310.00108/) (2023)
- [Progressive Visual Prompt Learning with Contrastive Feature Re-formation](https://mlsanity.com/paper/2304.08386/) (2023)
- [Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2403.16005/) (2024)
- [Towards Generalisable Video Moment Retrieval: Visual-Dynamic Injection to Image-Text Pre-Training](https://mlsanity.com/paper/2303.00040/) (2023)
- [MMSD2.0: Towards a Reliable Multi-modal Sarcasm Detection System](https://mlsanity.com/paper/2307.07135/) (2023)
- [SeD: Semantic-Aware Discriminator for Image Super-Resolution](https://mlsanity.com/paper/2402.19387/) (2024)
- [Bootstrapping Multi-view Representations for Fake News Detection](https://mlsanity.com/paper/2206.05741/) (2022)
- [Defense-Prefix for Preventing Typographic Attacks on CLIP](https://mlsanity.com/paper/2304.04512/) (2023)
- [Domain-Controlled Prompt Learning](https://mlsanity.com/paper/2310.07730/) (2023)
- [Unveiling Typographic Deceptions: Insights of the Typographic Vulnerability in Large Vision-Language Model](https://mlsanity.com/paper/2402.19150/) (2024)
- [Fine-grained Textual Inversion Network for Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2503.19296/) (2025)
- [LM-CPPF: Paraphrasing-Guided Data Augmentation for Contrastive Prompt-Based Few-Shot Fine-Tuning](https://mlsanity.com/paper/2305.18169/) (2023)
- [Multi-modal Fake News Detection on Social Media via Multi-grained Information Fusion](https://mlsanity.com/paper/2304.00827/) (2023)
- [CausalCLIPSeg: Unlocking CLIP's Potential in Referring Medical Image Segmentation with Causal Intervention](https://mlsanity.com/paper/2503.15949/) (2025)
- [Mixed Hierarchy Network for Image Restoration](https://mlsanity.com/paper/2302.09554/) (2023)
- [A3lign-DFER: Pioneering Comprehensive Dynamic Affective Alignment for Dynamic Facial Expression Recognition with CLIP](https://mlsanity.com/paper/2403.04294/) (2024)
- [GAMA: Generative Adversarial Multi-Object Scene Attacks](https://mlsanity.com/paper/2209.09502/) (2022)
- [FD-Align: Feature Discrimination Alignment for Fine-tuning Pre-Trained Models in Few-Shot Learning](https://mlsanity.com/paper/2310.15105/) (2023)
- [Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning](https://mlsanity.com/paper/2401.00701/) (2024)
- [ARTxAI: Explainable Artificial Intelligence Curates Deep Representation Learning for Artistic Images using Fuzzy Techniques](https://mlsanity.com/paper/2308.15284/) (2023)
- [MATE: Meet At The Embedding -- Connecting Images with Long Texts](https://mlsanity.com/paper/2407.09541/) (2024)
- [Have Large Vision-Language Models Mastered Art History?](https://mlsanity.com/paper/2409.03521/) (2024)
- [FER-former: Multi-modal Transformer for Facial Expression Recognition](https://mlsanity.com/paper/2303.12997/) (2023)
- [ArtRAG: Retrieval-Augmented Generation with Structured Context for Visual Art Understanding](https://mlsanity.com/paper/2505.06020/) (2025)
- [LEMoN: Label Error Detection using Multimodal Neighbors](https://mlsanity.com/paper/2407.18941/) (2024)
- [Generative AI in Agriculture: Creating Image Datasets Using DALL.E's Advanced Large Language Model Capabilities](https://mlsanity.com/paper/2307.08789/) (2023)

## Abstract (arXiv)

Existing computer vision research in artwork struggles with artwork's fine-grained attributes recognition and lack of curated annotated datasets due to their costly creation. To the best of our knowledge, we are one of the first methods to use CLIP (Contrastive Language-Image Pre-Training) to train a neural network on a variety of artwork images and text descriptions pairs. CLIP is able to learn directly from free-form art descriptions, or, if available, curated fine-grained labels. Model's zero-shot capability allows predicting accurate natural language description for a given image, without directly optimizing for the task. Our approach aims to solve 2 challenges: instance retrieval and fine-grained artwork attribute recognition. We use the iMet Dataset, which we consider the largest annotated artwork dataset. In this benchmark we achieved competitive results using only self-supervision.