mlsanityData to 7 Oct 2026

Vision-Language Models Do Not Understand Negation

arXiv 2501.09425 ↗First posted 16 Jan 2025cs.CV cs.CLcited by 69

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Many practical vision-language applications require models that understand negation, e.g., when using natural language to retrieve images which contain certain objects but not others. Despite advancements in vision-language models (VLMs) through large-scale training, their ability to comprehend negation remains underexplored. This study addresses the question: how well do current VLMs understand negation? We introduce NegBench, a new benchmark designed to evaluate negation understanding across 18 task variations and 79k examples spanning image, video, and medical datasets. The benchmark consists of two core tasks designed to evaluate negation understanding in diverse multimodal settings: Retrieval with Negation and Multiple Choice Questions with Negated Captions. Our evaluation reveals that modern VLMs struggle significantly with negation, often performing at chance level. To address these shortcomings, we explore a data-centric approach wherein we finetune CLIP models on large-scale synthetic datasets containing millions of negated captions. We show that this approach can result in a 10% increase in recall on negated queries and a 28% boost in accuracy on multiple-choice questions with negated captions.

Key facts. Cited by 69 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 54 references that are in the map. First posted on arXiv 16 Jan 2025.

Builds on

41 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negations29 Mar
  2. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?13 Dec
  3. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
  9. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  11. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  12. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
26 more
  1. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  2. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  3. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  4. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  5. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  6. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  7. 2019Multi-Stage Document Ranking with BERTcited by 200
  8. 2023Quilt-1M: One Million Image-Text Pairs for Histopathologycited by 192
  9. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  10. 2017Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenescited by 122
  11. 2018Learning Semantic Segmentation from Synthetic Data: A Geometrically Guided Input-Output Adaptation Approachcited by 119
  12. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  13. 2021Generative Models as a Data Source for Multiview Representation Learningcited by 104
  14. 2021CLIP-It! Language-Guided Video Summarizationcited by 99
  15. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  16. 2023CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Notcited by 79
  17. 2023Language models are not naysayers: An analysis of language models on negation benchmarkscited by 73
  18. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  19. 2024SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?cited by 64
  20. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understandingcited by 46
  21. 2022ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP Cuescited by 42
  22. 2023Real-Fake: Effective Training Data Synthesis Through Distribution Matchingcited by 35
  23. 2022Comprehending and Ordering Semantics for Image Captioningcited by 33
  24. 2022On Guiding Visual Attention with Language Specificationcited by 30
  25. 2023This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Modelscited by 18
  26. 2022FitCLIP: Refining Large-Scale Pretrained Image-Text Models for Zero-Shot Video Understanding Taskscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 31; 2026: 38.

69 papers.

Sort:

By area:

  1. 2025CLIP Behaves like a Bag-of-Words Model Cross-modally but not Uni-modallycited by 30
  2. 2025Is CLIP ideal? No. Can we fix it? Yes!cited by 20
  3. 2025Forgotten Polygons: Multimodal Large Language Models are Shape-Blindcited by 16
  4. 2025NegVQA: Can Vision Language Models Understand Negation?cited by 12
  5. 2025Do Large Language Models Truly Grasp Addition? A Rule-Focused Diagnostic Using Two-Integer Arithmeticcited by 8
  6. 2025CountGD++: Generalized Prompting for Open-World Countingcited by 8
  7. 2026AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answeringcited by 8
  8. 2025Negation: A Pink Elephant in the Large Language Models' Room?cited by 7
  9. 2025Temporal Concept Dynamics in Diffusion Models via Prompt-Conditioned Interventionscited by 7
  10. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluationcited by 7
  11. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?cited by 6
  12. 2025Don't Deceive Me: Mitigating Gaslighting through Attention Reallocation in LMMscited by 6
38 more
  1. 2025TNG-CLIP:Training-Time Negation Data Generation for Negation Awareness of CLIPcited by 6
  2. 2025Benchmarking Gaslighting Negation Attacks Against Reasoning Modelscited by 6
  3. 2025Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter Itcited by 6
  4. 2026Physics-IQ Verifiedcited by 6
  5. 2025Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understandingcited by 5
  6. 2026Watch Before You Answer: Learning from Visually Grounded Post-Trainingcited by 5
  7. 2025Negation-Aware Test-Time Adaptation for Vision-Language Modelscited by 4
  8. 2025A Comprehensive Taxonomy of Negation for NLP and Neural Retrieverscited by 4
  9. 2025The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation from Recognition to Reasoningcited by 4
  10. 2025Benchmarking Gaslighting Negation Attacks Against Multimodal Large Language Modelscited by 3
  11. 2025VSF: Simple, Efficient, and Effective Negative Guidance in Few-Step Image Generation Models By Value Sign Flipcited by 3
  12. 2025On the Brittleness of CLIP Text Encoderscited by 3
  13. 2025SpaceVLM: Sub-Space Modeling of Negation in Vision-Language Modelscited by 3
  14. 2026NEGATE: Constrained Semantic Guidance for Linguistic Negation in Text-to-Video Diffusioncited by 2
  15. 2026DEO: Training-Free Direct Embedding Optimization for Negation-Aware Retrievalcited by 2
  16. 2026RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observationcited by 2
  17. 2026Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMscited by 2
  18. 2026Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Modelscited by 2
  19. 2025What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Mergingcited by 1
  20. 2026Exploring SAIG Methods for an Objective Evaluation of XAIcited by 1
  21. 2026When Negation Is a Geometry Problem in Vision-Language Modelscited by 1
  22. 2026Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Modelscited by 1
  23. 2026DCR: Counterfactual Attractor Guidance for Rare Compositional Generationcited by 1
  24. 2026FlowCIR: Semantic Transport via Flow Matching for Zero-Shot Composed Image Retrievalcited by 1
  25. 2026The Sound of Absence: Audio-Language Embedding Models Struggle with Negationcited by 1
  26. 2026Learning to Detect Cross-Modal Negation: An Analysis of Latent Representations and an Attention-Based Solutioncited by 1
  27. 2026Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialoguecited by 1
  28. 2025An Iterative Feedback Mechanism for Improving Natural Language Class Descriptions in Open-Vocabulary Object Detectioncited by 0
  29. 2025Improving Physical Object State Representation in Text-to-Image Generative Systemscited by 0
  30. 2025Beyond Accuracy: Metrics that Uncover What Makes a 'Good' Visual Descriptorcited by 0
  31. 2025Response Wide Shut? Surprising Observations in Basic Vision Language Model Capabilitiescited by 0
  32. 2025Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMscited by 0
  33. 2025INQUIRE-Search: Interactive Discovery in Large-Scale Biodiversity Databasescited by 0
  34. 2025Adapting MLLMs for Nuanced Video Retrievalcited by 0
  35. 2026Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understandingcited by 0
  36. 2026Left-Right Symmetry Breaking in CLIP-style Vision-Language Models Trained on Synthetic Spatial-Relation Datacited by 0
  37. 2026Bi-MCQ: Reformulating Vision-Language Alignment for Negation Understandingcited by 0
  38. 2026NAST: Improving Negation Handling in Medical Vision-Language Models through Negation-Aware Selective Trainingcited by 0

50 most cited of 69 All 69 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Models
  2. 2025Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP
  3. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  4. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
  5. 2026HANCLIP: A Family of Hyperbolic Angular Negation Vision Language Models
  6. 2026Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding
  7. 2024NeIn: Telling What You Don't Want
  8. 2023Revisiting the Role of Language Priors in Vision-Language Models