mlsanityData to 7 Oct 2026

Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negations

arXiv 2403.20312 ↗First posted 29 Mar 2024cs.CVcited by 45

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Existing vision-language models (VLMs) treat text descriptions as a unit, confusing individual concepts in a prompt and impairing visual semantic matching and reasoning. An important aspect of reasoning in logic and language is negations. This paper highlights the limitations of popular VLMs such as CLIP, at understanding the implications of negations, i.e., the effect of the word "not" in a given prompt. To enable evaluation of VLMs on fluent prompts with negations, we present CC-Neg, a dataset containing 228,246 images, true captions and their corresponding negated captions. Using CC-Neg along with modifications to the contrastive loss of CLIP, our proposed CoN-CLIP framework, has an improved understanding of negations. This training paradigm improves CoN-CLIP's ability to encode semantics reliably, resulting in 3.85% average gain in top-1 accuracy for zero-shot image classification across 8 datasets. Further, CoN-CLIP outperforms CLIP on challenging compositionality benchmarks such as SugarCREPE by 4.4%, showcasing emergent compositional understanding of objects, relations, and attributes in text. Overall, our work addresses a crucial limitation of VLMs by introducing a dataset and framework that strengthens semantic associations between images and text, demonstrating improved large-scale foundation models with significantly reduced computational cost, promoting efficiency and accessibility.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 61 references that are in the map. First posted on arXiv 29 Mar 2024.

Builds on

39 of 61 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality26 Jun
  2. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?13 Dec
  3. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct

Other references · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
24 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  4. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  5. 2023PaLM 2 Technical Reportcited by 1,154
  6. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  7. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  8. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  9. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  10. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  11. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  12. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  13. 2023Improving CLIP Training with Language Rewritescited by 216
  14. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 182
  15. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  16. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  17. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  18. 2021CLIP-It! Language-Guided Video Summarizationcited by 99
  19. 2023Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statementscited by 44
  20. 2023OvarNet: Towards Open-vocabulary Object Attribute Recognitioncited by 43
  21. 2022ComCLIP: Training-Free Compositional Image and Text Matchingcited by 26
  22. 2020It's not a Non-Issue: Negation as a Source of Error in Machine Translationcited by 19
  23. 2020NegatER: Unsupervised Discovery of Negatives in Commonsense Knowledge Basescited by 9
  24. 2020Negation in Cognitive Reasoningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 23; 2026: 15.

45 papers.

Sort:

By area:

  1. 2024LamRA: Large Multimodal Model as Your Advanced Retrieval Assistantcited by 106
  2. 2025Vision-Language Models Do Not Understand Negationcited by 69
  3. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 57
  4. 2024INQUIRE: A Natural World Text-to-Image Retrieval Benchmarkcited by 33
  5. 2024Dynamic Negative Guidance of Diffusion Modelscited by 28
  6. 2025Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIPcited by 26
  7. 2025Is CLIP ideal? No. Can we fix it? Yes!cited by 20
  8. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  9. 2025Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Datacited by 12
  10. 2025NegVQA: Can Vision Language Models Understand Negation?cited by 12
  11. 2025Bringing CLIP to the Clinic: Dynamic Soft Labels and Negation-Aware Learning for Medical Analysiscited by 10
  12. 2025U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMscited by 10
33 more
  1. 2025Negation: A Pink Elephant in the Large Language Models' Room?cited by 7
  2. 2025TNG-CLIP:Training-Time Negation Data Generation for Negation Awareness of CLIPcited by 6
  3. 2025Advancing Compositional Awareness in CLIP with Efficient Fine-Tuningcited by 5
  4. 2025A Good CREPE needs more than just Sugar: Investigating Biases in Compositional Vision-Language Benchmarkscited by 5
  5. 2025Negation-Aware Test-Time Adaptation for Vision-Language Modelscited by 4
  6. 2024NeIn: Telling What You Don't Wantcited by 3
  7. 2025Benchmarking Gaslighting Negation Attacks Against Multimodal Large Language Modelscited by 3
  8. 2025Trade-offs in Image Generation: How Do Different Dimensions Interact?cited by 3
  9. 2025VSF: Simple, Efficient, and Effective Negative Guidance in Few-Step Image Generation Models By Value Sign Flipcited by 3
  10. 2025On the Brittleness of CLIP Text Encoderscited by 3
  11. 2026NEGATE: Constrained Semantic Guidance for Linguistic Negation in Text-to-Video Diffusioncited by 2
  12. 2026Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMscited by 2
  13. 2025From No to Know: Taxonomy, Challenges, and Opportunities for Negation Understanding in Multimodal Foundation Modelscited by 1
  14. 2025RB-FT: Rationale-Bootstrapped Fine-Tuning for Video Classificationcited by 1
  15. 2026Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understandingcited by 1
  16. 2026Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Modelscited by 1
  17. 2026DCR: Counterfactual Attractor Guidance for Rare Compositional Generationcited by 1
  18. 2024Learning Visual-Semantic Subspace Representationscited by 0
  19. 2025Dynamic Relation Inference via Verb Embeddingscited by 0
  20. 2025Improving Physical Object State Representation in Text-to-Image Generative Systemscited by 0
  21. 2025(Almost) Free Modality Stitching of Foundation Modelscited by 0
  22. 2025Native Hierarchical and Compositional Representations with Subspace Embeddingscited by 0
  23. 2025Contrastive vision-language learning with paraphrasing and negationcited by 0
  24. 2026Left-Right Symmetry Breaking in CLIP-style Vision-Language Models Trained on Synthetic Spatial-Relation Datacited by 0
  25. 2026Towards Effective Negation Modeling in Joint Audio-Text Models for Musiccited by 0
  26. 2026NAST: Improving Negation Handling in Medical Vision-Language Models through Negation-Aware Selective Trainingcited by 0
  27. 2026Mastering Negation: Boosting Grounding Models via Grouped Opposition-Based Learningcited by 0
  28. 2026No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Modelscited by 0
  29. 2026A Sanity Check on Composed Image Retrievalcited by 0
  30. 2026TPD: Temporal Prior Decoupling for Text-to-Video Diffusion Modelscited by 0
  31. 2026Benchmarking MLLMs via Cognitive Expected Scene Graph for Safety-Critical Visual Negation Understandingcited by 0
  32. 2026Absence is Presence: Understanding Visual Scene Negative Events Under Safety Cognitive Constraintcited by 0
  33. 2026Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When Negation Is a Geometry Problem in Vision-Language Models
  2. 2026Not Just What's There: Enabling CLIP to Comprehend Negated Visual Descriptions Without Fine-tuning
  3. 2025SpaceVLM: Sub-Space Modeling of Negation in Vision-Language Models
  4. 2025What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging
  5. 2026HANCLIP: A Family of Hyperbolic Angular Negation Vision Language Models
  6. 2026Disparities In Negation Understanding Across Languages In Vision-Language Models
  7. 2026Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
  8. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks