# CLIP Brings Better Features to Visual Aesthetics Learners

arXiv: https://arxiv.org/abs/2307.15640 · HTML: https://mlsanity.com/paper/2307.15640/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal distillation, one of 12 directions in Knowledge distillation.
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 28 Jul 2023.

## Placed in

- Knowledge distillation / [Multimodal distillation](https://mlsanity.com/distill/multimodal-distillation/)

## Builds on (most cited first, 29 of 29)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [NIMA: Neural Image Assessment](https://mlsanity.com/paper/1709.05424/) (2017)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective](https://mlsanity.com/paper/2303.14968/) (2023)
- [A Survey on Deep Semi-supervised Learning](https://mlsanity.com/paper/2103.00550/) (2021)
- [Photo Aesthetics Ranking Network with Attributes and Content Adaptation](https://mlsanity.com/paper/1606.01621/) (2016)
- [Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation](https://mlsanity.com/paper/2205.14141/) (2022)
- [VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining](https://mlsanity.com/paper/2303.14302/) (2023)
- [Personalized Image Aesthetics Assessment with Rich Attributes](https://mlsanity.com/paper/2203.16754/) (2022)
- [Effective Aesthetics Prediction with Multi-level Spatially Pooled Features](https://mlsanity.com/paper/1904.01382/) (2019)
- [A Closer Look at Self-Supervised Lightweight Vision Transformers](https://mlsanity.com/paper/2205.14443/) (2022)
- [Adaptive Fractional Dilated Convolution Network for Image Aesthetics Assessment](https://mlsanity.com/paper/2004.03015/) (2020)
- [Learned Perceptual Image Enhancement](https://mlsanity.com/paper/1712.02864/) (2017)
- [Distilling Knowledge from Object Classification to Aesthetics Assessment](https://mlsanity.com/paper/2206.00809/) (2022)
- [Composition and Style Attributes Guided Image Aesthetic Assessment](https://mlsanity.com/paper/2111.04647/) (2021)
- [A deep architecture for unified aesthetic prediction](https://mlsanity.com/paper/1708.04890/) (2017)

## Built on by (most cited first, 4 of 4)

- [Scaling Up Personalized Image Aesthetic Assessment via Task Vector Customization](https://mlsanity.com/paper/2407.07176/) (2024)
- [Understanding Implosion in Text-to-Image Generative Models](https://mlsanity.com/paper/2409.12314/) (2024)
- [On the rankability of visual embeddings](https://mlsanity.com/paper/2507.03683/) (2025)
- [Advancing Comprehensive Aesthetic Insight with Multi-Scale Text-Guided Self-Supervised Learning](https://mlsanity.com/paper/2412.11952/) (2024)

## Abstract (arXiv)

Image Aesthetics Assessment (IAA) is a challenging task due to its subjective nature and expensive manual annotations. Recent large-scale vision-language models, such as Contrastive Language-Image Pre-training (CLIP), have shown their promising representation capability for various downstream tasks. However, the application of CLIP to resource-constrained and low-data IAA tasks remains limited. While few attempts to leverage CLIP in IAA have mainly focused on carefully designed prompts, we extend beyond this by allowing models from different domains and with different model sizes to acquire knowledge from CLIP. To achieve this, we propose a unified and flexible two-phase CLIP-based Semi-supervised Knowledge Distillation (CSKD) paradigm, aiming to learn a lightweight IAA model while leveraging CLIP's strong generalization capability. Specifically, CSKD employs a feature alignment strategy to facilitate the distillation of heterogeneous CLIP teacher and IAA student models, effectively transferring valuable features from pre-trained visual representations to two lightweight IAA models, respectively. To efficiently adapt to downstream IAA tasks in a low-data regime, the two strong visual aesthetics learners then conduct distillation with unlabeled examples for refining and transferring the task-specific knowledge collaboratively. Extensive experiments demonstrate that the proposed CSKD achieves state-of-the-art performance on multiple widely used IAA benchmarks. Furthermore, analysis of attention distance and entropy before and after feature alignment shows the effective transfer of CLIP's feature representation to IAA models, which not only provides valuable guidance for the model initialization of IAA but also enhances the aesthetic feature representation of IAA models. Code will be made publicly available.