# Pruning the Paradox: How CLIP's Most Informative Heads Enhance Performance While Amplifying Bias

arXiv: https://arxiv.org/abs/2503.11103 · HTML: https://mlsanity.com/paper/2503.11103/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 48 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [On the Relationship between Self-Attention and Convolutional Layers](https://mlsanity.com/paper/1911.03584/) (2019)
- [Interpreting CLIP's Image Representation via Text-Based Decomposition](https://mlsanity.com/paper/2310.05916/) (2023)
- [Generating Visual Explanations](https://mlsanity.com/paper/1603.08507/) (2016)
- [Probing Image-Language Transformers for Verb Understanding](https://mlsanity.com/paper/2106.09141/) (2021)
- [CLIPood: Generalizing CLIP to Out-of-Distributions](https://mlsanity.com/paper/2302.00864/) (2023)
- [Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models](https://mlsanity.com/paper/2005.07310/) (2020)
- [DeAR: Debiasing Vision-Language Models with Additive Residuals](https://mlsanity.com/paper/2303.10431/) (2023)
- [ProtoPFormer: Concentrating on Prototypical Parts in Vision Transformers for Interpretable Image Recognition](https://mlsanity.com/paper/2208.10431/) (2022)
- [VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers](https://mlsanity.com/paper/2203.17247/) (2022)
- [SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples](https://mlsanity.com/paper/2312.00825/) (2023)
- [VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution](https://mlsanity.com/paper/2306.12424/) (2023)
- [Rosetta Neurons: Mining the Common Units in a Model Zoo](https://mlsanity.com/paper/2306.09346/) (2023)
- [Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps](https://mlsanity.com/paper/2204.11073/) (2022)
- [Interpreting the Second-Order Effects of Neurons in CLIP](https://mlsanity.com/paper/2406.04341/) (2024)

## Built on by (most cited first, 1 of 1)

- [Beyond Semantics: Disentangling Information Scope in Sparse Autoencoders for CLIP](https://mlsanity.com/paper/2604.05724/) (2026)

## Abstract (arXiv)

CLIP is one of the most popular foundation models and is heavily used for many vision-language tasks, yet little is known about its inner workings. As CLIP is increasingly deployed in real-world applications, it is becoming even more critical to understand its limitations and embedded social biases to mitigate potentially harmful downstream consequences. However, the question of what internal mechanisms drive both the impressive capabilities as well as problematic shortcomings of CLIP has largely remained unanswered. To bridge this gap, we study the conceptual consistency of text descriptions for attention heads in CLIP-like models. Specifically, we propose Concept Consistency Score (CCS), a novel interpretability metric that measures how consistently individual attention heads in CLIP models align with specific concepts. Our soft-pruning experiments reveal that high CCS heads are critical for preserving model performance, as pruning them leads to a significantly larger performance drop than pruning random or low CCS heads. Notably, we find that high CCS heads capture essential concepts and play a key role in out-of-domain detection, concept-specific reasoning, and video-language understanding. Moreover, we prove that high CCS heads learn spurious correlations which amplify social biases. These results position CCS as a powerful interpretability metric exposing the paradox of performance and social biases in CLIP models.