mlsanityData to 7 Oct 2026

Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP

arXiv 2501.10913 ↗First posted 19 Jan 2025cs.CV cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

While CLIP has significantly advanced multimodal understanding by bridging vision and language, the inability to grasp negation - such as failing to differentiate concepts like "parking" from "no parking" - poses substantial challenges. By analyzing the data used in the public CLIP model's pre-training, we posit this limitation stems from a lack of negation-inclusive data. To address this, we introduce data generation pipelines that employ a large language model (LLM) and a multimodal LLM to produce negation-inclusive captions. Fine-tuning CLIP with data generated from our pipelines, we develop NegationCLIP, which enhances negation awareness while preserving the generality. Moreover, to enable a comprehensive evaluation of negation understanding, we propose NegRefCOCOg-a benchmark tailored to test VLMs' ability to interpret negation across diverse expressions and positions within a sentence. Experiments on various CLIP architectures validate the effectiveness of our data generation pipelines in enhancing CLIP's ability to perceive negation accurately. Additionally, NegationCLIP's enhanced negation awareness has practical applications across various multimodal tasks, demonstrated by performance gains in text-to-image generation and referring image segmentation.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 46 of its 58 references that are in the map. First posted on arXiv 19 Jan 2025.

Builds on

46 of 58 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
34 more
  1. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  2. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2015VQA: Visual Question Answeringcited by 2,489
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  7. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  8. 2016Modeling Context in Referring Expressionscited by 1,054
  9. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  10. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  11. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  12. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  13. 2021Image Segmentation Using Text and Image Promptscited by 509
  14. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  15. 2022Perception Prioritized Training of Diffusion Modelscited by 301
  16. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Modelscited by 242
  17. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  18. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  19. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  20. 2023Alpha-CLIP: A CLIP Model Focusing on Wherever You Wantcited by 150
  21. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  22. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  23. 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116
  24. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  25. 2023Self-correcting LLM-controlled Diffusion Modelscited by 112
  26. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  27. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  28. 2024Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negationscited by 45
  29. 2019NegBERT: A Transfer Learning Approach for Negation Detection and Scope Resolutioncited by 36
  30. 2024FairerCLIP: Debiasing CLIP's Zero-Shot Predictions using Functions in RKHSscited by 25
  31. 2023This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Modelscited by 18
  32. 2022Learn to Understand Negation in Video Retrievalcited by 16
  33. 2024Toward Interactive Regional Understanding in Vision-Large Language Modelscited by 9
  34. 2023Active Open-Vocabulary Recognition: Let Intelligent Moving Mitigate CLIP Limitationscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 14.

26 papers.

Sort:

By area:

  1. 2025Multimodal Referring Segmentation: A Surveycited by 22
  2. 2025Logic Unseen: Revealing the Logical Blindspots of Vision-Language Modelscited by 7
  3. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluationcited by 7
  4. 2025TNG-CLIP:Training-Time Negation Data Generation for Negation Awareness of CLIPcited by 6
  5. 2025Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understandingcited by 5
  6. 2026Not Just What's There: Enabling CLIP to Comprehend Negated Visual Descriptions Without Fine-tuningcited by 5
  7. 2025Negation-Aware Test-Time Adaptation for Vision-Language Modelscited by 4
  8. 2025VSF: Simple, Efficient, and Effective Negative Guidance in Few-Step Image Generation Models By Value Sign Flipcited by 3
  9. 2025SpaceVLM: Sub-Space Modeling of Negation in Vision-Language Modelscited by 3
  10. 2025Early Estimation of Language to Latent Alignment in Diffusion Modelscited by 3
  11. 2026Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMscited by 2
  12. 2026Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Modelscited by 2
14 more
  1. 2025What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Mergingcited by 1
  2. 2025PowerCLIP: Powerset Alignment for Contrastive Pre-Trainingcited by 1
  3. 2026When Negation Is a Geometry Problem in Vision-Language Modelscited by 1
  4. 2026Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understandingcited by 1
  5. 2026The Truth Stays in the Family: Enhancing Contextual Grounding via Inherited Truthful Heads in Model Lineagescited by 1
  6. 2026Negation Beyond the Verbal Channel: Temporal Multimodal Correlates in Dialoguecited by 1
  7. 2025Hypothesis Testing in Imaging Inverse Problemscited by 0
  8. 2026Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understandingcited by 0
  9. 2026NAST: Improving Negation Handling in Medical Vision-Language Models through Negation-Aware Selective Trainingcited by 0
  10. 2026Concrete Jungle: Towards Concreteness Paved Contrastive Negative Mining for Compositional Understandingcited by 0
  11. 2026SENSE: Stereo OpEN Vocabulary SEmantic Segmentationcited by 0
  12. 2026What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Featurescited by 0
  13. 2026Benchmarking MLLMs via Cognitive Expected Scene Graph for Safety-Critical Visual Negation Understandingcited by 0
  14. 2026Absence is Presence: Understanding Visual Scene Negative Events Under Safety Cognitive Constraintcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision-Language Models Do Not Understand Negation
  2. 2026HANCLIP: A Family of Hyperbolic Angular Negation Vision Language Models
  3. 2026Disparities In Negation Understanding Across Languages In Vision-Language Models
  4. 2026CLIP Is Shortsighted: Paying Attention Beyond the First Sentence
  5. 2026Learning to Detect Cross-Modal Negation: An Analysis of Latent Representations and an Attention-Based Solution
  6. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions
  7. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  8. 2024Modeling Caption Diversity in Contrastive Vision-Language Pretraining