mlsanityData to 7 Oct 2026

When are Lemons Purple? The Concept Association Bias of Vision-Language Models

arXiv 2212.12043 ↗First posted 22 Dec 2022cs.CV cs.CL cs.LGcited by 23

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large-scale vision-language models such as CLIP have shown impressive performance on zero-shot image classification and image-to-text retrieval. However, such performance does not realize in tasks that require a finer-grained correspondence between vision and language, such as Visual Question Answering (VQA). As a potential cause of the difficulty of applying these models to VQA and similar tasks, we report an interesting phenomenon of vision-language models, which we call the Concept Association Bias (CAB). We find that models with CAB tend to treat input as a bag of concepts and attempt to fill in the other missing concept crossmodally, leading to an unexpected zero-shot prediction. We demonstrate CAB by showing that CLIP's zero-shot classification performance greatly suffers when there is a strong concept association between an object (e.g. eggplant) and an attribute (e.g. color purple). We also show that the strength of CAB predicts the performance on VQA. We observe that CAB is prevalent in vision-language models trained with contrastive losses, even when autoregressive losses are jointly employed. However, a model that solely relies on autoregressive loss seems to exhibit minimal or no signs of CAB.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 35 references that are in the map. First posted on arXiv 22 Dec 2022.

Builds on

27 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 26

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
14 more
  1. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  3. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  4. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  5. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  6. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  7. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  8. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  9. 2020On the Binding Problem in Artificial Neural Networkscited by 230
  10. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  11. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 101
  12. 2022DALLE-2 is Seeing Double: Flaws in Word-to-Concept Mapping in Text2Image Modelscited by 50
  13. 2020Rel3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3Dcited by 30
  14. 2021COVR: A test-bed for Visually Grounded Compositional Generalization with real imagescited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 10; 2026: 6.

23 papers.

Sort:

By area:

  1. 2024Long-CLIP: Unlocking the Long-Text Capability of CLIPcited by 298
  2. 2025AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIPcited by 72
  3. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  4. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 30
  5. 2024CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcyclingcited by 23
  6. 2025Is CLIP ideal? No. Can we fix it? Yes!cited by 20
  7. 2024Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Functioncited by 19
  8. 2025Object-centric Binding in Contrastive Language-Image Pretrainingcited by 12
  9. 2025HalLoc: Token-level Localization of Hallucinations for Vision Language Modelscited by 9
  10. 2025Common Data Properties Limit Object-Attribute Binding in CLIPcited by 4
  11. 2024AttriPrompter: Auto-Prompting with Attribute Semantics for Zero-shot Nuclei Detection via Visual-Language Pre-trained Modelscited by 3
  12. 2024Automatic benchmarking of large multimodal models via iterative experiment programmingcited by 2
11 more
  1. 2024Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledgecited by 2
  2. 2025A Reality Check of Vision-Language Pre-training in Radiology: Have We Progressed Using Text?cited by 2
  3. 2025LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentationcited by 1
  4. 2025FairT2I: Mitigating Social Bias in Text-to-Image Generation via Large Language Model-Assisted Detection and Attribute Rebalancingcited by 0
  5. 2025Benchmarking Attribute Discrimination in Infant-Scale Vision-Language Modelscited by 0
  6. 2026Half-Truths Break Similarity-Based Retrievalcited by 0
  7. 2026No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Modelscited by 0
  8. 2026PGT: Procedurally Generated Tasks for improving visual grounding in MLLMscited by 0
  9. 2026How can embedding models bind concepts?cited by 0
  10. 2026Intermediate Text Representation Guided Text-to-Image Generation for Enhancing One-and-Only Alignmentcited by 0
  11. 2026How Do Diffusion Classifiers Decide? A Bias-Centric Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Neglected Tails in Vision-Language Models
  2. 2025Benchmarking and Mitigating MCQA Selection Bias of Large Vision-Language Models
  3. 2023Revisiting the Role of Language Priors in Vision-Language Models
  4. 2025Can VLMs Recall Factual Associations From Visual References?
  5. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  6. 2022VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models
  7. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features
  8. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models