# Don't Judge Before You CLIP: A Unified Approach for Perceptual Tasks

arXiv: https://arxiv.org/abs/2503.13260 · HTML: https://mlsanity.com/paper/2503.13260/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 61 references that are in the map.
- First posted on arXiv 17 Mar 2025.

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [MUSIQ: Multi-scale Image Quality Transformer](https://mlsanity.com/paper/2108.05997/) (2021)
- [Exploring CLIP for Assessing the Look and Feel of Images](https://mlsanity.com/paper/2207.12396/) (2022)
- [KonIQ-10k: An ecologically valid database for deep learning of blind image quality assessment](https://mlsanity.com/paper/1910.06180/) (2019)
- [Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective](https://mlsanity.com/paper/2303.14968/) (2023)
- [From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Quality](https://mlsanity.com/paper/1912.10088/) (2019)
- [Massive Online Crowdsourced Study of Subjective and Objective Picture Quality](https://mlsanity.com/paper/1511.02919/) (2015)
- [No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency](https://mlsanity.com/paper/2108.06858/) (2021)
- [Re-IQA: Unsupervised Learning for Image Quality Assessment in the Wild](https://mlsanity.com/paper/2304.00451/) (2023)
- [EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes](https://mlsanity.com/paper/2307.07961/) (2023)
- [Transformer for Image Quality Assessment](https://mlsanity.com/paper/2101.01097/) (2020)
- [Data-Efficient Image Quality Assessment with Attention-Panel Decoder](https://mlsanity.com/paper/2304.04952/) (2023)
- [MDAN: Multi-level Dependent Attention Network for Visual Emotion Analysis](https://mlsanity.com/paper/2203.13443/) (2022)
- [Stimuli-Aware Visual Emotion Analysis](https://mlsanity.com/paper/2109.01812/) (2021)
- [SOLVER: Scene-Object Interrelated Visual Emotion Reasoning Network](https://mlsanity.com/paper/2110.12334/) (2021)
- [AMNet: Memorability Estimation with Attention](https://mlsanity.com/paper/1804.03115/) (2018)
- [Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion](https://mlsanity.com/paper/2307.13463/) (2023)
- [Embracing New Techniques in Deep Learning for Estimating Image Memorability](https://mlsanity.com/paper/2105.10598/) (2021)
- [Learning to Compose Diversified Prompts for Image Emotion Classification](https://mlsanity.com/paper/2201.10963/) (2022)
- [Image Memorability Prediction with Vision Transformers](https://mlsanity.com/paper/2301.08647/) (2023)
- [fMRI Exploration of Visual Quality Assessment](https://mlsanity.com/paper/2404.18162/) (2024)

## Built on by (most cited first, 2 of 2)

- [How to Take a Memorable Picture? Empowering Users with Actionable Feedback](https://mlsanity.com/paper/2602.21877/) (2026)
- [A Lightweight Global-Target Framework for Multi-Domain No-Reference Image Quality Assessment in UAV Imagery](https://mlsanity.com/paper/2604.13112/) (2026)

## Abstract (arXiv)

Visual perceptual tasks aim to predict human judgment of images (e.g., emotions invoked by images, image quality assessment). Unlike objective tasks such as object/scene recognition, perceptual tasks rely on subjective human assessments, making its data-labeling difficult. The scarcity of such human-annotated data results in small datasets leading to poor generalization. Typically, specialized models were designed for each perceptual task, tailored to its unique characteristics and its own training dataset. We propose a unified architectural framework for solving multiple different perceptual tasks leveraging CLIP as a prior. Our approach is based on recent cognitive findings which indicate that CLIP correlates well with human judgment. While CLIP was explicitly trained to align images and text, it implicitly also learned human inclinations. We attribute this to the inclusion of human-written image captions in CLIP's training data, which contain not only factual image descriptions, but inevitably also human sentiments and emotions. This makes CLIP a particularly strong prior for perceptual tasks. Accordingly, we suggest that minimal adaptation of CLIP suffices for solving a variety of perceptual tasks. Our simple unified framework employs a lightweight adaptation to fine-tune CLIP to each task, without requiring any task-specific architectural changes. We evaluate our approach on three tasks: (i) Image Memorability Prediction, (ii) No-reference Image Quality Assessment, and (iii) Visual Emotion Analysis. Our model achieves state-of-the-art results on all three tasks, while demonstrating improved generalization across different datasets.