mlsanityData to 14 Oct 2026

Know-Thy-Not: Training-Free Targeted Negation in Dual-Encoder Vision-Language Models

arXiv 2610.17440 ↗First posted 12 Oct 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP also fits: VLM benchmarks and visual reasoning

Abstract (arXiv)

Dual-encoder vision-language models systematically fail at negation, scoring ''a cat without a dog'' nearly identically to ''a cat with a dog.'' We show that this failure is inherent to the single-embedding framework. Under contrastive training, single-embedding methods can at best ignore the negated concept (dilution) but never penalize its presence (rejection). We propose Know-Thy-Not (KTN), a training-free method that introduces an image-side rejection dimension to penalize the negated concept directly in the visual representation. We also introduce COCO-Confusion, the first benchmark requiring genuine rejection to succeed. Across 9 backbones, KTN improves COCO-Confusion mAP by up to +25.6 over vanilla VLMs and +9.6 over the strongest existing method.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 30 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

27 of 30 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Vision-Language Models Do Not Understand Negation16 Jan
  2. 2024Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negations29 Mar
  3. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality26 Jun
  4. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?13 Dec
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?4 Oct

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
10 more
  1. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  2. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  3. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  4. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  5. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  6. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  7. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  8. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
  9. 2025Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIPcited by 28
  10. 2025Is CLIP ideal? No. Can we fix it? Yes!cited by 21

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies
  2. 2024Unveiling Encoder-Free Vision-Language Models
  3. 2026LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives
  4. 2025Perception Encoder: The best visual embeddings are not at the output of the network
  5. 2025SpaceVLM: Sub-Space Modeling of Negation in Vision-Language Models
  6. 2022Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training
  7. 2026Conceptualizing Embeddings: Sparse Disentanglement for Vision-Language Models
  8. 2022Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics