# IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models

arXiv: https://arxiv.org/abs/2403.15952 · HTML: https://mlsanity.com/paper/2403.15952/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 22 of its 65 references that are in the map.
- First posted on arXiv 23 Mar 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022)
- [What the DAAM: Interpreting Stable Diffusion Using Cross Attention](https://mlsanity.com/paper/2210.04885/) (2022)
- [Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images](https://mlsanity.com/paper/2303.07274/) (2023)
- [GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration](https://mlsanity.com/paper/2311.12015/) (2023)
- [Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models](https://mlsanity.com/paper/2402.12058/) (2024)
- [What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance](https://mlsanity.com/paper/1912.06960/) (2019)
- [Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?](https://mlsanity.com/paper/2311.00047/) (2023)

## Built on by (most cited first, 30 of 31)

- [Vision Language Models are Biased](https://mlsanity.com/paper/2505.23941/) (2025)
- [Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly](https://mlsanity.com/paper/2406.10638/) (2024)
- [ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustness](https://mlsanity.com/paper/2504.10514/) (2025)
- [M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance](https://mlsanity.com/paper/2502.18778/) (2025)
- [VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models](https://mlsanity.com/paper/2503.07478/) (2025)
- [Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words](https://mlsanity.com/paper/2501.06254/) (2025)
- [MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs](https://mlsanity.com/paper/2511.14159/) (2025)
- [Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMs](https://mlsanity.com/paper/2502.16435/) (2025)
- [VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors](https://mlsanity.com/paper/2604.02486/) (2026)
- [IllusionBench+: A Large-scale and Comprehensive Benchmark for Visual Illusion Understanding in Vision-Language Models](https://mlsanity.com/paper/2501.00848/) (2025)
- [When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding](https://mlsanity.com/paper/2506.05551/) (2025)
- [The Art of Deception: Color Visual Illusions and Diffusion Models](https://mlsanity.com/paper/2412.10122/) (2024)
- [HueManity: Probing Fine-Grained Visual Perception in MLLMs](https://mlsanity.com/paper/2506.03194/) (2025)
- [Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusions](https://mlsanity.com/paper/2601.22150/) (2026)
- [Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models](https://mlsanity.com/paper/2602.21704/) (2026)
- [Efficient Multi-modal Long Context Learning for Training-free Adaptation](https://mlsanity.com/paper/2505.19812/) (2025)
- [FineGRAIN: Evaluating Failure Modes of Text-to-Image Models with Vision Language Model Judges](https://mlsanity.com/paper/2512.02161/) (2025)
- [Quantum-tunnelling deep neural network for optical illusion recognition](https://mlsanity.com/paper/2407.11013/) (2024)
- [Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusions](https://mlsanity.com/paper/2507.22617/) (2025)
- [Do Large Vision-Language Models Distinguish between the Actual and Apparent Features of Illusions?](https://mlsanity.com/paper/2506.05765/) (2025)
- [Illusions in Humans and AI: How Visual Perception Aligns and Diverges](https://mlsanity.com/paper/2508.12422/) (2025)
- [The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs](https://mlsanity.com/paper/2512.15949/) (2025)
- [Empowering Multimodal LLMs with External Tools: A Comprehensive Survey](https://mlsanity.com/paper/2508.10955/) (2025)
- [Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward](https://mlsanity.com/paper/2604.04500/) (2026)
- [ChainMPQ: Interleaved Text-Image Reasoning Chains for Mitigating Relation Hallucinations](https://mlsanity.com/paper/2510.06292/) (2025)
- [ChromouVQA: Benchmarking Vision-Language Models under Chromatic Camouflaged Images](https://mlsanity.com/paper/2512.05137/) (2025)
- [Illusion-Aware Visual Preprocessing and Anti-Illusion Prompting for Classic Illusion Understanding in Vision-Language Models](https://mlsanity.com/paper/2605.08841/) (2026)
- [Task-Aware Structured Memory for Dynamic Multi-modal In-Context Learning](https://mlsanity.com/paper/2606.11853/) (2026)
- [Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities](https://mlsanity.com/paper/2607.27747/) (2026)
- [Look Where You Say You're Looking: Self-Grounded Attention for Visual Reasoning](https://mlsanity.com/paper/2610.05023/) (2026)

## Abstract (arXiv)

The advent of Vision Language Models (VLM) has allowed researchers to investigate the visual understanding of a neural network using natural language. Beyond object classification and detection, VLMs are capable of visual comprehension and common-sense reasoning. This naturally led to the question: How do VLMs respond when the image itself is inherently unreasonable? To this end, we present IllusionVQA: a diverse dataset of challenging optical illusions and hard-to-interpret scenes to test the capability of VLMs in two distinct multiple-choice VQA tasks - comprehension and soft localization. GPT4V, the best performing VLM, achieves 62.99% accuracy (4-shot) on the comprehension task and 49.7% on the localization task (4-shot and Chain-of-Thought). Human evaluation reveals that humans achieve 91.03% and 100% accuracy in comprehension and localization. We discover that In-Context Learning (ICL) and Chain-of-Thought reasoning substantially degrade the performance of Gemini-Pro in the localization task. Tangentially, we discover a potential weakness in the ICL capabilities of VLMs: they fail to locate optical illusions even when the correct answer is in the context window as a few-shot example.