# Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions

arXiv: https://arxiv.org/abs/2412.08169 · HTML: https://mlsanity.com/paper/2412.08169/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images](https://mlsanity.com/paper/2303.07274/) (2023)
- [Visual Anagrams: Generating Multi-View Optical Illusions with Diffusion Models](https://mlsanity.com/paper/2311.17919/) (2023)
- [Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?](https://mlsanity.com/paper/2311.00047/) (2023)
- [Diffusion Illusions: Hiding Images in Plain Sight](https://mlsanity.com/paper/2312.03817/) (2023)
- [CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models](https://mlsanity.com/paper/2402.13607/) (2024)
- [Evaluating Vision-Language Models on Bistable Images](https://mlsanity.com/paper/2405.19423/) (2024)
- [ImageNet-trained deep neural network exhibits illusion-like response to the Scintillating Grid](https://mlsanity.com/paper/1907.09019/) (2019)

## Built on by (most cited first, 11 of 11)

- [Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies](https://mlsanity.com/paper/2602.01816/) (2026)
- [SemVink: Advancing VLMs' Semantic Understanding of Optical Illusions via Visual Global Thinking](https://mlsanity.com/paper/2506.02803/) (2025)
- [Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusions](https://mlsanity.com/paper/2507.22617/) (2025)
- [MUCAR: Benchmarking Multilingual Cross-Modal Ambiguity Resolution for Multimodal Large Language Models](https://mlsanity.com/paper/2506.17046/) (2025)
- [The Perceptual Observatory Characterizing Robustness and Grounding in MLLMs](https://mlsanity.com/paper/2512.15949/) (2025)
- [Now You See the Hate: Adaptive View Retrieval for Hidden Hateful Illusions](https://mlsanity.com/paper/2607.19061/) (2026)
- [Spatial Colour Mixing Illusions as a Perception Stress Test for Vision-Language Models](https://mlsanity.com/paper/2603.06141/) (2026)
- [Illusion-Aware Visual Preprocessing and Anti-Illusion Prompting for Classic Illusion Understanding in Vision-Language Models](https://mlsanity.com/paper/2605.08841/) (2026)
- [Evaluating Chinese Ambiguity Understanding in Large Language Models](https://mlsanity.com/paper/2605.15635/) (2026)
- [Position: The Systemic Lack of Agency in Visual Reasoning](https://mlsanity.com/paper/2606.14795/) (2026)
- [When MLLMs miss what humans read: diagnosing and learning to read squint text](https://mlsanity.com/paper/2610.13465/) (2026)

## Abstract (arXiv)

In recent years, Visual Question Answering (VQA) has made significant strides, particularly with the advent of multimodal models that integrate vision and language understanding. However, existing VQA datasets often overlook the complexities introduced by image illusions, which pose unique challenges for both human perception and model interpretation. In this study, we introduce a novel task called Illusory VQA, along with four specialized datasets: IllusionMNIST, IllusionFashionMNIST, IllusionAnimals, and IllusionChar. These datasets are designed to evaluate the performance of state-of-the-art multimodal models in recognizing and interpreting visual illusions. We assess the zero-shot performance of various models, fine-tune selected models on our datasets, and propose a simple yet effective solution for illusion detection using Gaussian and blur low-pass filters. We show that this method increases the performance of models significantly and in the case of BLIP-2 on IllusionAnimals without any fine-tuning, it outperforms humans. Our findings highlight the disparity between human and model perception of illusions and demonstrate that fine-tuning and specific preprocessing techniques can significantly enhance model robustness. This work contributes to the development of more human-like visual understanding in multimodal models and suggests future directions for adapting filters using learnable parameters.