mlsanityData to 7 Oct 2026

Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans?

arXiv 2311.00047 ↗First posted 31 Oct 2023cs.AI cs.CL cs.CVcited by 32

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) are trained on vast amounts of data captured by humans emulating our understanding of the world. However, known as visual illusions, human's perception of reality isn't always faithful to the physical world. This raises a key question: do VLMs have the similar kind of illusions as humans do, or do they faithfully learn to represent reality? To investigate this question, we build a dataset containing five types of visual illusions and formulate four tasks to examine visual illusions in state-of-the-art VLMs. Our findings have shown that although the overall alignment is low, larger models are closer to human perception and more susceptible to visual illusions. Our dataset and initial findings will promote a better understanding of visual illusions in humans and machines and provide a stepping stone for future computational models that can better align humans and machines in perceiving and communicating about the shared visual world. The code and data are available at https://github.com/vl-illusion/dataset.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 8 of its 37 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

8 of 37 references.

References · 8

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  6. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  7. 2022VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformerscited by 53
  8. 2019What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performancecited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 10; 2026: 12.

32 papers.

Sort:

By area:

  1. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 58
  2. 2024Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Modelscited by 55
  3. 2023Visual cognition in multimodal large language modelscited by 38
  4. 2024IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Modelscited by 29
  5. 2025IllusionBench+: A Large-scale and Comprehensive Benchmark for Visual Illusion Understanding in Vision-Language Modelscited by 12
  6. 2025Human Cognitive Benchmarks Reveal Foundational Visual Gaps in MLLMscited by 12
  7. 2024Social Perception of Faces in a Vision-Language Modelcited by 10
  8. 2024Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusionscited by 10
  9. 2024VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuningcited by 9
  10. 2024The Art of Deception: Color Visual Illusions and Diffusion Modelscited by 9
  11. 2024MindSet: Vision. A toolbox for testing DNNs on key psychological experimentscited by 8
  12. 2026Do VLMs Perceive or Recall? Probing Visual Perception vs. Memory with Classic Visual Illusionscited by 8
20 more
  1. 2026Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomaliescited by 8
  2. 2024Evaluating Vision-Language Models on Bistable Imagescited by 6
  3. 2025Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusionscited by 6
  4. 2025Testing the Limits of Fine-Tuning for Improving Visual Cognition in Vision Language Modelscited by 4
  5. 2025Do Large Vision-Language Models Distinguish between the Actual and Apparent Features of Illusions?cited by 3
  6. 2025Illusions in Humans and AI: How Visual Perception Aligns and Divergescited by 3
  7. 2025The Perceptual Observatory Characterizing Robustness and Grounding in MLLMscited by 3
  8. 2025Visual Language Models show widespread visual deficits on neuropsychological testscited by 2
  9. 2026SMSP: A Plug-and-Play Strategy of Multi-Scale Perception for MLLMs to Perceive Visual Illusionscited by 2
  10. 2025ChromouVQA: Benchmarking Vision-Language Models under Chromatic Camouflaged Imagescited by 1
  11. 2026Now You See the Hate: Adaptive View Retrieval for Hidden Hateful Illusionscited by 1
  12. 2025A biological vision inspired framework for machine perception of abutting grating illusory contourscited by 0
  13. 2026Anchored, Not Graded: Vision-Language Models Fail at Slant-from-Texture Perceptioncited by 0
  14. 2026Readable Yet Unpredictable: Rotated-Outcome Prediction in Vision-Language Modelscited by 0
  15. 2026Denoising Models Develop Human-Like Perceptual Illusion Representations Across Architecturescited by 0
  16. 2026Can LVLMs Uncover the Truth Behind Visual Illusions? An Analysis of Perceptual and Reasoning Capabilitiescited by 0
  17. 2026More Accurate, Less Human: Gestalt Grouping in Vision Modelscited by 0
  18. 2026Beyond What Meets the Eye: Unveiling Situational Illusions for Multimodal Large Language Modelscited by 0
  19. 2026Auditory Illusion Benchmark for Large Audio Language Modelscited by 0
  20. 2026(How) Do MLLMs Report Bistable Images Like Humans?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating Model Perception of Color Illusions in Photorealistic Scenes
  2. 2025SemVink: Advancing VLMs' Semantic Understanding of Optical Illusions via Visual Global Thinking
  3. 2024The Illusion-Illusion: Vision Language Models See Illusions Where There Are None
  4. 2026Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models
  5. 2026Illusion-Aware Visual Preprocessing and Anti-Illusion Prompting for Classic Illusion Understanding in Vision-Language Models
  6. 2024Can We Talk Models Into Seeing the World Differently?
  7. 2026Seeing the Evidence, Missing the Answer: Tool-Guided Vision-Language Models on Visual Illusions
  8. 2025Not There Yet: Evaluating Vision Language Models in Simulating the Visual Perception of People with Low Vision