# Mirage Probes: How Vision Models Fake Visual Understanding

arXiv: https://arxiv.org/abs/2606.13870 · HTML: https://mlsanity.com/paper/2606.13870/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 36 of its 39 references that are in the map.
- First posted on arXiv 11 Jun 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 36)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://mlsanity.com/paper/2309.14525/) (2023)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [OCR-free Document Understanding Transformer](https://mlsanity.com/paper/2111.15664/) (2021)
- [Med-Flamingo: a Multimodal Medical Few-shot Learner](https://mlsanity.com/paper/2307.15189/) (2023)
- [Hallucination of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2404.18930/) (2024)
- [A Survey on Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2402.00253/) (2024)
- [PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering](https://mlsanity.com/paper/2305.10415/) (2023)
- [MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding](https://mlsanity.com/paper/2501.18362/) (2025)
- [Ovis: Structural Embedding Alignment for Multimodal Large Language Model](https://mlsanity.com/paper/2405.20797/) (2024)
- [Hallucination Augmented Contrastive Learning for Multimodal Large Language Model](https://mlsanity.com/paper/2312.06968/) (2023)
- [Detecting Strategic Deception Using Linear Probes](https://mlsanity.com/paper/2502.03407/) (2025)

## Built on by (most cited first, 5 of 5)

- [Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes](https://mlsanity.com/paper/2608.05000/) (2026)
- [Beyond Frame Selection: Generative Latent Evidence Aggregation for Long-Video Understanding](https://mlsanity.com/paper/2607.28516/) (2026)
- [Unsupervised Features Mining via Activation Geometry](https://mlsanity.com/paper/2607.04222/) (2026)
- [Don't Repeat Yourself: Stopping Verbatim Loops at Sampling Time](https://mlsanity.com/paper/2608.22761/) (2026)
- [Teaching Vision-Language Models to Decline: Support Completion Against Mirage](https://mlsanity.com/paper/2610.16465/) (2026)

## Abstract (arXiv)

Vision-language models (VLMs) can answer image-based questions confidently, and often correctly, even when no image is provided. This mirage behavior inflates benchmark scores without reflecting visual grounding. Prior work treats this as a single failure mode. We argue it is two. Using Mirage Probes, a contrastive probing framework that pairs paraphrased question variants with matched mirage and non-mirage labels on the same image, we show that mirage behavior is linearly decodable from internal activations across residual stream, MLP, post-attention, and attention-head sites in two open-source VLMs. We demonstrate that a Naive Bayes text baseline cannot recover this signal, ruling out surface lexical confounds. Cross-benchmark separability patterns, together with a novel Prior Harnessing Index (PHI) measuring how much a model can answer from text alone, expose two distinct regimes: textual biases, where the model answers from language priors without engaging visual representations, and spurious images, where it constructs false visual content in latent space and answers as if grounded. The distinction has direct mitigation consequences: text-distribution cleaning can address the first regime but cannot reach the second, since spurious-image mirages live in the model's visual representations rather than its text. Faithful visual grounding will require interventions at the representational level.