mlsanityData to 7 Oct 2026

Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications

arXiv 2108.02818 ↗First posted 5 Aug 2021cs.CV cs.AI cs.CYcited by 128

Not on the map. Why?

Cited by 51 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, there have been breakthroughs in computer vision ("CV") models that are more generalizable with the advent of models such as CLIP and ALIGN. In this paper, we analyze CLIP and highlight some of the challenges such models pose. CLIP reduces the need for task specific training data, potentially opening up many niche tasks to automation. CLIP also allows its users to flexibly specify image classification classes in natural language, which we find can shift how biases manifest. Additionally, through some preliminary probes we find that CLIP can inherit biases found in prior computer vision systems. Given the wide and unpredictable domain of uses for such models, this raises questions regarding what sufficiently safe behaviour for such systems may look like. These results add evidence to the growing body of work calling for a change in the notion of a 'better' model--to move beyond simply looking at higher accuracy at task-oriented capability evaluations, and towards a broader 'better' that takes into account deployment-critical features such as different use contexts, and people who interact with the model when thinking about model deployment.

Key facts. Cited by 128 papers in the map (data through 7 Oct 2026). Builds on 7 of its 23 references that are in the map. First posted on arXiv 5 Aug 2021.

Builds on

7 of 23 references.

References · 7

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  5. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  6. 2021A Straightforward Framework For Video Retrieval Using CLIPcited by 66
  7. 2021About Face: A Survey of Facial Recognition Evaluationcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 17; 2023: 33; 2024: 39; 2025: 23; 2026: 12.

128 papers.

Sort:

By area:

  1. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  2. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  3. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  4. 2024An Introduction to Vision-Language Modelingcited by 175
  5. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  6. 2023Debiasing Vision-Language Models via Biased Promptscited by 147
  7. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  8. 2022A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learningcited by 104
  9. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  10. 2024MoReVQA: Exploring Modular Reasoning Models for Video Question Answeringcited by 74
  11. 2023Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignmentcited by 71
  12. 2023ITI-GEN: Inclusive Text-to-Image Generationcited by 64
38 more
  1. 2024The Neglected Tails in Vision-Language Modelscited by 63
  2. 2023Discovering and Mitigating Visual Biases through Keyword Explanationcited by 58
  3. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  4. 2022A CLIP-Hitchhiker's Guide to Long Video Retrievalcited by 57
  5. 2023PIGEON: Predicting Image Geolocationscited by 55
  6. 2024Examining Gender and Racial Bias in Large Vision-Language Models Using a Novel Dataset of Parallel Imagescited by 55
  7. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 55
  8. 2023VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolutioncited by 52
  9. 2023Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Featurescited by 52
  10. 2023Mitigating Spurious Correlations in Multi-modal Models during Fine-tuningcited by 47
  11. 2023Modular Visual Question Answering via Code Generationcited by 47
  12. 2022Contrastive Language-Vision AI Models Pretrained on Web-Scraped Multimodal Data Exhibit Sexual Objectification Biascited by 43
  13. 2022American == White in Multimodal Language-and-Image AIcited by 39
  14. 2024KVQ: Kwai Video Quality Assessment for Short-form Videoscited by 38
  15. 2023Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparitiescited by 31
  16. 2024GenderBias-VL: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probingcited by 31
  17. 2022Evidence for Hypodescent in Visual Semantic AIcited by 30
  18. 2022Markedness in Visual Semantic AIcited by 29
  19. 2024AI-Face: A Million-Scale Demographically Annotated AI-Generated Face Dataset and Fairness Benchmarkcited by 28
  20. 2023Balancing the Picture: Debiasing Vision-Language Datasets with Synthetic Contrast Setscited by 27
  21. 2023Survey of Social Bias in Vision-Language Modelscited by 26
  22. 2023A Friendly Face: Do Text-to-Image Systems Rely on Stereotypes when the Input is Under-Specified?cited by 25
  23. 2024FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSscited by 25
  24. 2023Gender Biases in Automatic Evaluation Metrics for Image Captioningcited by 24
  25. 2023What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentationcited by 24
  26. 2023MultiModal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision Language Modelscited by 23
  27. 2023Auditing Gender Presentation Differences in Text-to-Image Modelscited by 22
  28. 2022FairCLIP: Social Bias Elimination based on Attribute Prototype Learning and Representation Neutralizationcited by 20
  29. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  30. 2024A Unified Debiasing Approach for Vision-Language Models across Modalities and Taskscited by 18
  31. 2024BendVLM: Test-Time Debiasing of Vision-Language Embeddingscited by 17
  32. 2024Joint Vision-Language Social Bias Removal for CLIPcited by 17
  33. 2021Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformationcited by 16
  34. 2022Neural Eigenfunctions Are Structured Representation Learnerscited by 16
  35. 2023PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contextscited by 16
  36. 2023Parrot Captions Teach CLIP to Spot Textcited by 16
  37. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Modelscited by 16
  38. 2024Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help Youcited by 15

50 most cited of 128 All 128 (JSON)