mlsanityData to 7 Oct 2026

IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models

arXiv 2403.15952 ↗First posted 23 Mar 2024cs.CV cs.CLcited by 29

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The advent of Vision Language Models (VLM) has allowed researchers to investigate the visual understanding of a neural network using natural language. Beyond object classification and detection, VLMs are capable of visual comprehension and common-sense reasoning. This naturally led to the question: How do VLMs respond when the image itself is inherently unreasonable? To this end, we present IllusionVQA: a diverse dataset of challenging optical illusions and hard-to-interpret scenes to test the capability of VLMs in two distinct multiple-choice VQA tasks - comprehension and soft localization. GPT4V, the best performing VLM, achieves 62.99% accuracy (4-shot) on the comprehension task and 49.7% on the localization task (4-shot and Chain-of-Thought). Human evaluation reveals that humans achieve 91.03% and 100% accuracy in comprehension and localization. We discover that In-Context Learning (ICL) and Chain-of-Thought reasoning substantially degrade the performance of Gemini-Pro in the localization task. Tangentially, we discover a potential weakness in the ICL capabilities of VLMs: they fail to locate optical illusions even when the correct answer is in the context window as a few-shot example.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 65 references that are in the map. First posted on arXiv 23 Mar 2024.

Builds on

22 of 65 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?31 Oct

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
9 more
  1. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  3. 2023Larger language models do in-context learning differentlycited by 386
  4. 2022DETRs with Collaborative Hybrid Assignments Trainingcited by 273
  5. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 233
  6. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  7. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76
  8. 2024Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Modelscited by 56
  9. 2019What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performancecited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 19; 2026: 7.

29 papers.

Sort:

By area:

  1. 2025Vision Language Models are Biasedcited by 63
  2. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 29
  3. 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustnesscited by 28
  4. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  5. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  6. 2025Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Wordscited by 17
  7. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 14
  8. 2025IllusionBench+: A Large-scale and Comprehensive Benchmark for Visual Illusion Understanding in Vision-Language Modelscited by 12
  9. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 12
  10. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  11. 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchorscited by 12
  12. 2024The Art of Deception: Color Visual Illusions and Diffusion Modelscited by 9
17 more
  1. 2025HueManity: Probing Fine-Grained Visual Perception in MLLMscited by 9
  2. 2026Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusionscited by 8
  3. 2025Efficient Multi-modal Long Context Learning for Training-free Adaptationcited by 7
  4. 2024Quantum-tunnelling deep neural network for optical illusion recognitioncited by 6
  5. 2025Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusionscited by 6
  6. 2025FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judgescited by 6
  7. 2026Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Modelscited by 6
  8. 2025Do Large Vision-Language Models Distinguish between the Actual and Apparent Features of Illusions?cited by 3
  9. 2025Illusions in Humans and AI: How Visual Perception Aligns and Divergescited by 3
  10. 2025The Perceptual Observatory Characterizing Robustness and Grounding in MLLMscited by 3
  11. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  12. 2025ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinationscited by 1
  13. 2025ChromouVQA: Benchmarking Vision-Language Models under Chromatic Camouflaged Imagescited by 1
  14. 2026Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Rewardcited by 1
  15. 2026Illusion-Aware Visual Preprocessing and Anti-Illusion Prompting for Classic Illusion Understanding in Vision-Language Modelscited by 0
  16. 2026Task-Aware Structured Memory for Dynamic Multi-modal In-Context Learningcited by 0
  17. 2026Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing the Evidence, Missing the Answer: Tool-Guided Vision-Language Models on Visual Illusions
  2. 2025SemVink: Advancing VLMs' Semantic Understanding of Optical Illusions via Visual Global Thinking
  3. 2026Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies
  4. 2026Beyond Shortcuts: Mitigating Visual Illusions in Frozen VLMs via Qualitative Reasoning
  5. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models
  6. 2025VLMs have Tunnel Vision: Evaluating Nonlocal Visual Reasoning in Leading VLMs
  7. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
  8. 2025Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models