# Color in Visual-Language Models: CLIP deficiencies

arXiv: https://arxiv.org/abs/2502.04470 · HTML: https://mlsanity.com/paper/2502.04470/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 1 of its 15 references that are in the map.
- First posted on arXiv 6 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 1 of 1)

- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)

## Built on by (most cited first, 4 of 4)

- [Color encoding in Latent Space of Stable Diffusion Models](https://mlsanity.com/paper/2512.09477/) (2025)
- [Color Names in Vision-Language Models](https://mlsanity.com/paper/2509.22524/) (2025)
- [Lang2Motion: Bridging Language and Motion through Joint Embedding Spaces](https://mlsanity.com/paper/2512.10617/) (2025)
- [Feature Competition and Its Representational Origins in Vision Language Models: Evidence from the Stroop Paradigm](https://mlsanity.com/paper/2610.12650/) (2026)

## Abstract (arXiv)

This work explores how color is encoded in CLIP (Contrastive Language-Image Pre-training) which is currently the most influential VML (Visual Language model) in Artificial Intelligence. After performing different experiments on synthetic datasets created for this task, we conclude that CLIP is able to attribute correct color labels to colored visual stimulus, but, we come across two main deficiencies: (a) a clear bias on achromatic stimuli that are poorly related to the color concept, thus white, gray and black are rarely assigned as color labels; and (b) the tendency to prioritize text over other visual information. Here we prove it is highly significant in color labelling through an exhaustive Stroop-effect test. With the aim to find the causes of these color deficiencies, we analyse the internal representation at the neuron level. We conclude that CLIP presents an important amount of neurons selective to text, specially in deepest layers of the network, and a smaller amount of multi-modal color neurons which could be the key of understanding the concept of color properly. Our investigation underscores the necessity of refining color representation mechanisms in neural networks to foster a more comprehensive comprehension of colors as humans understand them, thereby advancing the efficacy and versatility of multimodal models like CLIP in real-world scenarios.