# What's Holding Back Latent Visual Reasoning?

arXiv: https://arxiv.org/abs/2605.18445 · HTML: https://mlsanity.com/paper/2605.18445/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 18 May 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 18 of 18)

- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [BLINK: Multimodal Large Language Models Can See but Not Perceive](https://mlsanity.com/paper/2404.12390/) (2024)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://mlsanity.com/paper/2507.01006/) (2025)
- [VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models](https://mlsanity.com/paper/2504.15279/) (2025)
- [Latent Visual Reasoning](https://mlsanity.com/paper/2509.24251/) (2025)
- [Monet: Reasoning in Latent Visual Space Beyond Images and Language](https://mlsanity.com/paper/2511.21395/) (2025)
- [LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?](https://mlsanity.com/paper/2503.19990/) (2025)
- [VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs](https://mlsanity.com/paper/2407.01863/) (2024)
- [Interleaved Latent Visual Reasoning with Selective Perceptual Modeling](https://mlsanity.com/paper/2512.05665/) (2025)
- [Reasoning in the Dark: Interleaved Vision-Text Reasoning in Latent Space](https://mlsanity.com/paper/2510.12603/) (2025)
- [Do Latent Tokens Think? A Causal and Adversarial Analysis of Chain-of-Continuous-Thought](https://mlsanity.com/paper/2512.21711/) (2025)
- [Forest Before Trees: Latent Superposition for Efficient Visual Reasoning](https://mlsanity.com/paper/2601.06803/) (2026)
- [Vision-aligned Latent Reasoning for Multi-modal Large Language Model](https://mlsanity.com/paper/2602.04476/) (2026)
- [Imagination Helps Visual Reasoning, But Not Yet in Latent Space](https://mlsanity.com/paper/2602.22766/) (2026)
- [LanteRn: Latent Visual Structured Reasoning](https://mlsanity.com/paper/2603.25629/) (2026)

## Built on by (most cited first, 7 of 7)

- [Beyond Visual Memory: Mechanistic Diagnostics of Latent Visual Reasoning](https://mlsanity.com/paper/2606.01287/) (2026)
- [Reason Through the Latent! Making Latent Visual Reasoning Necessary](https://mlsanity.com/paper/2609.06746/) (2026)
- [GLaQ: Grounding Latent Queries in Visual Evidence for Multimodal Reasoning](https://mlsanity.com/paper/2608.15517/) (2026)
- [Uni-LaDiR: Latent Diffusion Unifies Multimodal Reasoning](https://mlsanity.com/paper/2609.19878/) (2026)
- [Rethinking Latent Visual Reasoning: Grounding Latent Reasoning in Visual Evidence](https://mlsanity.com/paper/2609.34563/) (2026)
- [WM-VLM: Probing Internal World Models for Interleaved Visual-Textual Reasoning](https://mlsanity.com/paper/2609.34826/) (2026)
- [GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Models](https://mlsanity.com/paper/2610.18694/) (2026)

## Abstract (arXiv)

Humans can approach complex visual problems by mentally simulating intermediate visual steps, rather than reasoning through language alone. Inspired by this, several works on Vision-Language Models have recently explored chain-of-thought reasoning with continuous latent tokens as intermediate visual imagination steps. In this work, we investigate how recent models leverage such latent tokens. Surprisingly, we find that model accuracy is unaffected when latent tokens are replaced by uninformative dummy tokens. This indicates that latent tokens play a minimal causal role in the model's final prediction. To better understand this phenomenon, we analyze both the training signal provided by oracle latent representations and the quality of the latent tokens generated at inference time. Our experiments reveal two crucial issues holding back latent visual reasoning: First, in most existing datasets, oracle latent tokens provide limited additional information beyond the original image and do not substantially simplify the task, leading models to ignore them during training and effectively bypassing them at inference time. When fine-tuned on a diagnostic dataset, in which latent tokens provide sufficient support for the final prediction, we show that models can causally rely on them. Second, the latent tokens produced at inference time deviate from their corresponding oracle representations, collapsing to a narrow region and preventing benefits even when the model relies on them. Overall, our findings suggest that future progress in latent visual reasoning depends on two key pillars: high-quality datasets with informative intermediate steps and more precise latent token prediction.