mlsanityData to 7 Oct 2026

If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions

arXiv 2403.16442 ↗First posted 25 Mar 2024cs.CL cs.CV cs.LGcited by 14

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent works often assume that Vision-Language Model (VLM) representations are based on visual attributes like shape. However, it is unclear to what extent VLMs prioritize this information to represent concepts. We propose Extract and Explore (EX2), a novel approach to characterize textual features that are important for VLMs. EX2 uses reinforcement learning to align a large language model with VLM preferences and generates descriptions that incorporate features that are important for the VLM. Then, we inspect the descriptions to identify features that contribute to VLM representations. Using EX2, we find that spurious descriptions have a major role in VLM representations despite providing no helpful information, e.g., Click to enlarge photo of CONCEPT. More importantly, among informative descriptions, VLMs rely significantly on non-visual attributes like habitat (e.g., North America) to represent visual concepts. Also, our analysis reveals that different VLMs prioritize different attributes in their representations. Overall, we show that VLMs do not simply match images to scene descriptions and that non-visual or even spurious descriptions significantly influence their representations.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 70 references that are in the map. First posted on arXiv 25 Mar 2024.

Builds on

46 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Probing Conceptual Understanding of Large Visual-Language Models7 Apr

Other references · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  12. 2020Learning to summarize from human feedbackcited by 2,562
33 more
  1. 2019Language Models as Knowledge Bases?cited by 1,922
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  3. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  4. 2019How Can We Know What Language Models Know?cited by 953
  5. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  6. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  7. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  8. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  9. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  10. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 310
  11. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  12. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  13. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  14. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  15. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  16. 2022Linearly Mapping from Image to Text Spacecited by 139
  17. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  18. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  19. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 101
  20. 2023Waffling around for Performance: Visual Classification with Random Words and Broad Conceptscited by 100
  21. 2023CHiLS: Zero-Shot Image Classification with Hierarchical Label Setscited by 79
  22. 2023An Inverse Scaling Law for CLIP Trainingcited by 76
  23. 2022What do Vision Transformers Learn? A Visual Explorationcited by 75
  24. 2023Learning Concise and Descriptive Attributes for Visual Recognitioncited by 73
  25. 2020Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressionscited by 33
  26. 2022Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?cited by 31
  27. 2021Does Vision-and-Language Pretraining Improve Lexical Grounding?cited by 25
  28. 2023DesCo: Learning Object Recognition with Rich Language Descriptionscited by 25
  29. 2022Unit Testing for Concepts in Neural Networkscited by 23
  30. 2023Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificitycited by 14
  31. 2023Follow-Up Differential Descriptions: Language Models Resolve Ambiguities for Image Classificationcited by 13
  32. 2024What do we learn from inverting CLIP models?cited by 13
  33. 2023Text Descriptions are Compressive and Invariant Representations for Visual Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 3.

14 papers.

Sort:

By area:

  1. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 30
  2. 2024Improving Concept Alignment in Vision-Language Concept Bottleneck Modelscited by 4
  3. 2025FLOSS: Free Lunch in Open-vocabulary Semantic Segmentationcited by 4
  4. 2025CIVET: Systematic Evaluation of Understanding in VLMscited by 4
  5. 2025Attribute-formed Class-specific Concept Space: Endowing Language Bottleneck Model with Better Interpretability and Scalabilitycited by 3
  6. 2024Test-time Contrastive Concepts for Open-world Semantic Segmentation with Vision-Language Modelscited by 2
  7. 2024Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Modelscited by 2
  8. 2026UniCVR: From Alignment to Reranking for Unified Zero-Shot Composed Visual Retrievalcited by 2
  9. 2026Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLMcited by 1
  10. 2025Do I look like a `cat.n.01` to you? A Taxonomy Image Generation Benchmarkcited by 0
  11. 2025An Iterative Feedback Mechanism for Improving Natural Language Class Descriptions in Open-Vocabulary Object Detectioncited by 0
  12. 2025AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decompositioncited by 0
2 more
  1. 2025Contrastive Learning with Enhanced Abstract Representations using Grouped Loss of Abstract Semantic Supervisioncited by 0
  2. 2026IGLOSS: Image Generation for Lidar Open-vocabulary Semantic Segmentationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors
  2. 2025Hidden in plain sight: VLMs overlook their visual representations
  3. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models
  4. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  5. 2024Towards Interpreting Visual Information Processing in Vision-Language Models
  6. 2024Can We Talk Models Into Seeing the World Differently?
  7. 2024An Introduction to Vision-Language Modeling
  8. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict