# Democratizing Contrastive Language-Image Pre-training: A CLIP Benchmark of Data, Model, and Supervision

arXiv: https://arxiv.org/abs/2203.05796 · HTML: https://mlsanity.com/paper/2203.05796/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 31 references that are in the map.
- First posted on arXiv 11 Mar 2022.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Show and Tell: A Neural Image Caption Generator](https://mlsanity.com/paper/1411.4555/) (2014)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://mlsanity.com/paper/2111.07991/) (2021)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [SLIP: Self-supervision meets Language-Image Pre-training](https://mlsanity.com/paper/2112.12750/) (2021)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [CLIP2Video: Mastering Video-Text Retrieval via Image CLIP](https://mlsanity.com/paper/2106.11097/) (2021)
- [Simple but Effective: CLIP Embeddings for Embodied AI](https://mlsanity.com/paper/2111.09888/) (2021)
- [Combined Scaling for Zero-shot Transfer Learning](https://mlsanity.com/paper/2111.10050/) (2021)
- [CLIP4Caption: CLIP for Video Caption](https://mlsanity.com/paper/2110.06615/) (2021)
- [Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation](https://mlsanity.com/paper/2112.09445/) (2021)
- [INTERN: A New Learning Paradigm Towards General Vision](https://mlsanity.com/paper/2111.08687/) (2021)

## Built on by (most cited first, 30 of 36)

- [MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretraining](https://mlsanity.com/paper/2208.12262/) (2022)
- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining](https://mlsanity.com/paper/2204.14095/) (2022)
- [MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training](https://mlsanity.com/paper/2311.17049/) (2023)
- [Perceptual Grouping in Contrastive Vision-Language Models](https://mlsanity.com/paper/2210.09996/) (2022)
- [ALIP: Adaptive Language-Image Pre-training with Synthetic Caption](https://mlsanity.com/paper/2308.08428/) (2023)
- [UniCLIP: Unified Framework for Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2209.13430/) (2022)
- [Learning to Localize Objects Improves Spatial Reasoning in Visual-LLMs](https://mlsanity.com/paper/2404.07449/) (2024)
- [FLAIR: VLM with Fine-grained Language-informed Image Representations](https://mlsanity.com/paper/2412.03561/) (2024)
- [HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention](https://mlsanity.com/paper/2303.02995/) (2023)
- [Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokens](https://mlsanity.com/paper/2303.14865/) (2023)
- [Vocabulary-free Image Classification](https://mlsanity.com/paper/2306.00917/) (2023)
- [Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentors](https://mlsanity.com/paper/2211.13224/) (2022)
- [Learning Visual Representations via Language-Guided Sampling](https://mlsanity.com/paper/2302.12248/) (2023)
- [NLIP: Noise-robust Language-Image Pre-training](https://mlsanity.com/paper/2212.07086/) (2022)
- [Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2303.11866/) (2023)
- [COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training](https://mlsanity.com/paper/2412.01814/) (2024)
- [ProAPO: Progressively Automatic Prompt Optimization for Visual Classification](https://mlsanity.com/paper/2502.19844/) (2025)
- [FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resources](https://mlsanity.com/paper/2407.01445/) (2024)
- [Vocabulary-free Image Classification and Semantic Segmentation](https://mlsanity.com/paper/2404.10864/) (2024)
- [FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language models](https://mlsanity.com/paper/2405.10286/) (2024)
- [Hybrid Distillation: Connecting Masked Autoencoders with Contrastive Learners](https://mlsanity.com/paper/2306.15876/) (2023)
- [MLIP: Efficient Multi-Perspective Language-Image Pretraining with Exhaustive Data Utilization](https://mlsanity.com/paper/2406.01460/) (2024)
- [ProtoCLIP: Prototypical Contrastive Language Image Pretraining](https://mlsanity.com/paper/2206.10996/) (2022)
- [Retrieval-based Disentangled Representation Learning with Natural Language Supervision](https://mlsanity.com/paper/2212.07699/) (2022)
- [LightCLIP: Learning Multi-Level Interaction for Lightweight Vision-Language Models](https://mlsanity.com/paper/2312.00674/) (2023)
- [MobileViCLIP: An Efficient Video-Text Model for Mobile Devices](https://mlsanity.com/paper/2508.07312/) (2025)
- [Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition](https://mlsanity.com/paper/2406.09388/) (2024)
- [Benchmarking Vision-Language Contrastive Methods for Medical Representation Learning](https://mlsanity.com/paper/2406.07450/) (2024)
- [MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Data](https://mlsanity.com/paper/2603.08202/) (2026)

## Abstract (arXiv)

Contrastive Language-Image Pretraining (CLIP) has emerged as a novel paradigm to learn visual models from language supervision. While researchers continue to push the frontier of CLIP, reproducing these works remains challenging. This is because researchers do not choose consistent training recipes and even use different data, hampering the fair comparison between different methods. In this work, we propose CLIP-benchmark, a first attempt to evaluate, analyze, and benchmark CLIP and its variants. We conduct a comprehensive analysis of three key factors: data, supervision, and model architecture. We find considerable intuitive or counter-intuitive insights: (1). Data quality has a significant impact on performance. (2). Certain supervision has different effects for Convolutional Networks (ConvNets) and Vision Transformers (ViT). Applying more proper supervision can effectively improve the performance of CLIP. (3). Curtailing the text encoder reduces the training cost but not much affect the final performance. Moreover, we further combine DeCLIP with FILIP, bringing us the strongest variant DeFILIP. The CLIP-benchmark would be released at: https://github.com/Sense-GVT/DeCLIP for future CLIP research.