# The Illusion-Illusion: Vision Language Models See Illusions Where There Are None

arXiv: https://arxiv.org/abs/2412.18613 · HTML: https://mlsanity.com/paper/2412.18613/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 6 of its 28 references that are in the map.
- First posted on arXiv 7 Dec 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 6 of 6)

- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [Easy Problems That LLMs Get Wrong](https://mlsanity.com/paper/2405.19616/) (2024)

## Built on by (most cited first, 14 of 14)

- [Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding](https://mlsanity.com/paper/2502.11492/) (2025)
- [Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusions](https://mlsanity.com/paper/2601.22150/) (2026)
- [Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Models](https://mlsanity.com/paper/2502.15678/) (2025)
- [Do Large Vision-Language Models Distinguish between the Actual and Apparent Features of Illusions?](https://mlsanity.com/paper/2506.05765/) (2025)
- [Illusions in Humans and AI: How Visual Perception Aligns and Diverges](https://mlsanity.com/paper/2508.12422/) (2025)
- [ChromouVQA: Benchmarking Vision-Language Models under Chromatic Camouflaged Images](https://mlsanity.com/paper/2512.05137/) (2025)
- [CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models](https://mlsanity.com/paper/2605.17826/) (2026)
- [Now You See the Hate: Adaptive View Retrieval for Hidden Hateful Illusions](https://mlsanity.com/paper/2607.19061/) (2026)
- [What's in Common? Multimodal Models Hallucinate When Reasoning Across Scenes](https://mlsanity.com/paper/2511.03768/) (2025)
- [Are they human? Detecting large language models by probing human memory constraints](https://mlsanity.com/paper/2604.00016/) (2026)
- [The Riddle Riddle: Testing Flexible Reasoning in Large Language Models and Humans](https://mlsanity.com/paper/2606.27103/) (2026)
- [Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilities](https://mlsanity.com/paper/2607.27747/) (2026)
- [(How) Do MLLMs Report Bistable Images Like Humans?](https://mlsanity.com/paper/2609.13254/) (2026)
- [Cheap, open agents make LLM pollution harder to mitigate](https://mlsanity.com/paper/2609.31054/) (2026)

## Abstract (arXiv)

Illusions are entertaining, but they are also a useful diagnostic tool in cognitive science, philosophy, and neuroscience. A typical illusion shows a gap between how something `really is' and how something `appears to be', and this gap helps us understand the mental processing that led to how something appears to be. Illusions are also useful for investigating artificial systems, and much research has examined whether computational models of perception fall prey to the same illusions as people. Here, I invert the standard use of perceptual illusions to examine basic processing errors in current vision language models. I present these models with illusory-illusions, neighbors of common illusions that should not elicit processing errors. These include such things as perfectly reasonable ducks, crooked lines that truly are crooked, circles that seem to have different sizes because they are, in fact, of different sizes, and so on. I show that many current vision language systems mistakenly see these illusion-illusions as illusions. I suggest that such failures are part of broader failures already discussed in the literature.