# IllusionBench+: A Large-scale and Comprehensive Benchmark for Visual Illusion Understanding in Vision-Language Models

arXiv: https://arxiv.org/abs/2501.00848 · HTML: https://mlsanity.com/paper/2501.00848/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 33 references that are in the map.
- First posted on arXiv 1 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 14 of 14)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [CogVLM: Visual Expert for Pretrained Language Models](https://mlsanity.com/paper/2311.03079/) (2023)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://mlsanity.com/paper/2401.16420/) (2024)
- [Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?](https://mlsanity.com/paper/2311.00047/) (2023)
- [IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models](https://mlsanity.com/paper/2403.15952/) (2024)
- [Neural Networks Trained on Natural Scenes Exhibit Gestalt Closure](https://mlsanity.com/paper/1903.01069/) (2019)

## Built on by (most cited first, 12 of 12)

- [MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs](https://mlsanity.com/paper/2511.14159/) (2025)
- [Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusions](https://mlsanity.com/paper/2601.22150/) (2026)
- [SemVink: Advancing VLMs' Semantic Understanding of Optical Illusions via Visual Global Thinking](https://mlsanity.com/paper/2506.02803/) (2025)
- [Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusions](https://mlsanity.com/paper/2507.22617/) (2025)
- [Illusions in Humans and AI: How Visual Perception Aligns and Diverges](https://mlsanity.com/paper/2508.12422/) (2025)
- [SMSP: A Plug-and-Play Strategy of Multi-Scale Perception for MLLMs to Perceive Visual Illusions](https://mlsanity.com/paper/2603.23118/) (2026)
- [Learn to Think: Improving Multimodal Reasoning through Vision-Aware Self-Improvement Training](https://mlsanity.com/paper/2605.11931/) (2026)
- [System-Mediated Attention Imbalances Make Vision-Language Models Say Yes](https://mlsanity.com/paper/2601.12430/) (2026)
- [Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models](https://mlsanity.com/paper/2603.06141/) (2026)
- [FruitEnsemble: MLLM-Guided Arbitration for Heterogeneous ensemble in Fine-Grained Fruit Recognition](https://mlsanity.com/paper/2605.20892/) (2026)
- [Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities](https://mlsanity.com/paper/2607.27747/) (2026)
- [Auditory Illusion Benchmark for Large Audio Language Models](https://mlsanity.com/paper/2609.02277/) (2026)

## Abstract (arXiv)

Current Visual Language Models (VLMs) show impressive image understanding but struggle with visual illusions, especially in real-world scenarios. Existing benchmarks focus on classical cognitive illusions, which have been learned by state-of-the-art (SOTA) VLMs, revealing issues such as hallucinations and limited perceptual abilities. To address this gap, we introduce IllusionBench, a comprehensive visual illusion dataset that encompasses not only classic cognitive illusions but also real-world scene illusions. This dataset features 1,051 images, 5,548 question-answer pairs, and 1,051 golden text descriptions that address the presence, causes, and content of the illusions. We evaluate ten SOTA VLMs on this dataset using true-or-false, multiple-choice, and open-ended tasks. In addition to real-world illusions, we design trap illusions that resemble classical patterns but differ in reality, highlighting hallucination issues in SOTA models. The top-performing model, GPT-4o, achieves 80.59% accuracy on true-or-false tasks and 76.75% on multiple-choice questions, but still lags behind human performance. In the semantic description task, GPT-4o's hallucinations on classical illusions result in low scores for trap illusions, even falling behind some open-source models. IllusionBench is, to the best of our knowledge, the largest and most comprehensive benchmark for visual illusions in VLMs to date.