mlsanityData to 14 Oct 2026

NegVQA: Can Vision Language Models Understand Negation?

arXiv 2505.22946 ↗First posted 28 May 2025cs.CL cs.AI cs.CVcited by 13

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Negation is a fundamental linguistic phenomenon that can entirely reverse the meaning of a sentence. As vision language models (VLMs) continue to advance and are deployed in high-stakes applications, assessing their ability to comprehend negation becomes essential. To address this, we introduce NegVQA, a visual question answering (VQA) benchmark consisting of 7,379 two-choice questions covering diverse negation scenarios and image-question distributions. We construct NegVQA by leveraging large language models to generate negated versions of questions from existing VQA datasets. Evaluating 20 state-of-the-art VLMs across seven model families, we find that these models struggle significantly with negation, exhibiting a substantial performance drop compared to their responses to the original questions. Furthermore, we uncover a U-shaped scaling trend, where increasing model size initially degrades performance on NegVQA before leading to improvements. Our benchmark reveals critical gaps in VLMs' negation understanding and offers insights into future VLM development. Project page available at https://yuhui-zh15.github.io/NegVQA/.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 48 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

37 of 48 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation6 Jan
  2. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  3. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 33

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  7. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
21 more
  1. 2019Towards VQA Models That Can Readcited by 1,718
  2. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  3. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  5. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  6. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  7. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  8. 2016A Diagram Is Worth A Dozen Imagescited by 804
  9. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  10. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  11. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  12. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  13. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 243
  14. 2023Inverse Scaling: When Bigger Isn't Bettercited by 207
  15. 2024Observational Scaling Laws and the Predictability of Language Model Performancecited by 130
  16. 2022Inverse scaling can become U-shapedcited by 76
  17. 2025Vision-Language Models Do Not Understand Negationcited by 72
  18. 2024TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domainscited by 69
  19. 2024Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negationscited by 46
  20. 2024How and where does CLIP process negation?cited by 16
  21. 2023Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Modelscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 10.

13 papers.

Sort:

By area:

  1. 2026Limited Linguistic Diversity in Embodied AI Datasetscited by 6
  2. 2025The Artificial Intelligence Cognitive Examination: A Survey on the Evolution of Multimodal Evaluation from Recognition to Reasoningcited by 4
  3. 2025How Well Do Vision--Language Models Understand Cities? A Comparative Study on Spatial Reasoning from Street-View Imagescited by 2
  4. 2026NEGATE: Constrained Semantic Guidance for Linguistic Negation in Text-to-Video Diffusioncited by 2
  5. 2026Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMscited by 2
  6. 2026Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Modelscited by 2
  7. 2026DCR: Counterfactual Attractor Guidance for Rare Compositional Generationcited by 1
  8. 2026The Sound of Absence: Audio-Language Embedding Models Struggle with Negationcited by 1
  9. 2025Native Hierarchical and Compositional Representations with Subspace Embeddingscited by 0
  10. 2026NAST: Improving Negation Handling in Medical Vision-Language Models through Negation-Aware Selective Trainingcited by 0
  11. 2026FuzzingRL: Reinforcement Fuzz-Testing for Revealing VLM Failurescited by 0
  12. 2026Absence is Presence: Understanding Visual Scene Negative Events Under Safety Cognitive Constraintcited by 0
1 more
  1. 2026NegT2IBench: When Negation Changes the Picture. A Polarity Benchmark for Text-to-Image Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Skeleton-and-Strategy Prompting: Training-Free Negation Understanding for Vision-Language Models
  2. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  3. 2024A Comprehensive Survey on Visual Question Answering Datasets and Algorithms
  4. 2025The Quest for Visual Understanding: A Journey Through the Evolution of Visual Question Answering
  5. 2023Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts
  6. 2025Exploring Advanced Techniques for Visual Question Answering: A Comprehensive Comparison
  7. 2023Exploring Question Decomposition for Zero-Shot VQA
  8. 2024Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types