mlsanityData to 14 Oct 2026

The Hard Positive Truth about Vision-Language Compositionality

arXiv 2409.17958 ↗First posted 26 Sep 2024cs.CL cs.CVcited by 22

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Several benchmarks have concluded that our best vision-language models (e.g., CLIP) are lacking in compositionality. Given an image, these benchmarks probe a model's ability to identify its associated caption amongst a set of compositional distractors. In response, a surge of recent proposals show improvements by finetuning CLIP with distractors as hard negatives. Our investigations reveal that these improvements have, in fact, been significantly overstated -- because existing benchmarks do not probe whether finetuned vision-language models remain invariant to hard positives. By curating an evaluation dataset with 112,382 hard negatives and hard positives, we uncover that including hard positives decreases CLIP's performance by 12.9%, while humans perform effortlessly at 99%. CLIP finetuned with hard negatives results in an even larger decrease, up to 38.7%. With this finding, we then produce a 1,775,259 image-text training set with both hard negative and hard positive captions. By training with both, we see improvements on existing benchmarks while simultaneously improving performance on hard positives, indicating a more robust improvement in compositionality. Our work suggests the need for future research to rigorously test and improve CLIP's understanding of semantic relationships between related "positive" concepts.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 52 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

40 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality26 Jun
  2. 2023Image Captioners Are Scalable Vision Learners Too13 Jun
  3. 2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality23 May
  4. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?13 Dec
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Segment Anythingcited by 8,234
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  10. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  11. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  12. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
23 more
  1. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  2. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  3. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  4. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  5. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  6. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  7. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  8. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  9. 2016Visual Relationship Detection with Language Priorscited by 334
  10. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  11. 2019Compositionality decomposed: how do neural networks generalise?cited by 265
  12. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  13. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  14. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 250
  15. 2023Improving CLIP Training with Language Rewritescited by 217
  16. 2020Adversarial Filters of Dataset Biasescited by 160
  17. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  18. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
  19. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  20. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  21. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  22. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
  23. 2023Fighting Bias with Bias: Promoting Model Robustness by Amplifying Dataset Biasescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 7; 2026: 14.

22 papers.

Sort:

By area:

  1. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 31
  2. 2025Decoupled Global-Local Alignment for Improving Compositional Understandingcited by 9
  3. 2024Natural Language Inference Improves Compositionality in Vision-Language Modelscited by 8
  4. 2026AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answeringcited by 8
  5. 2025New Dataset and Methods for Fine-Grained Compositional Referring Expression Comprehension via Specialist-MLLM Collaborationcited by 7
  6. 2025A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarkscited by 5
  7. 2025Importance Sampling for Multi-Negative Multimodal Direct Preference Optimizationcited by 5
  8. 2026OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understandingcited by 5
  9. 2025VAQUUM: Are Vague Quantifiers Grounded in Visual Data?cited by 4
  10. 2025The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation from Recognition to Reasoningcited by 4
  11. 2026Global Geometry Is Not Enough for Vision Representationscited by 2
  12. 2026Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Modelscited by 2
10 more
  1. 2026Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understandingcited by 1
  2. 2026Cross-Modal Masked Compositional Concept Modeling for Enhancing Visio-Linguistic Compositionalitycited by 1
  3. 2026Explicit Logic Channel for Validation and Enhancement of MLLMs on Zero-Shot Taskscited by 0
  4. 2026Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understandingcited by 0
  5. 2026Foveated Probes Recover Localized Binding Information in Vision Foundation Modelscited by 0
  6. 2026PragMatch: Separating Pragmatic Incongruity from Cross-Modal Mismatch in Large Vision-Language Modelscited by 0
  7. 2026RePair: Turning Retrieval Failures into Counterfactual Hard Pairscited by 0
  8. 2026BindCLIP: One Balanced Coupling For Compositional Vision Language Scoringcited by 0
  9. 2026Diagnosing the Sources of Compositional Failure in Vision-Language Models: A Controlled Analysiscited by 0
  10. 2026The Value of Being Wrong: Self-Mined Visual In-Context Learning from Errorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  2. 2023Text encoders bottleneck compositionality in contrastive vision-language models
  3. 2024Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations
  4. 2024Toward a Holistic Evaluation of Robustness in CLIP Models
  5. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives
  6. 2024Language Plays a Pivotal Role in the Object-Attribute Compositional Generalization of CLIP
  7. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models
  8. 2026Language-Conditioned Visual Grounding with CLIP Multilingual