# Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought

arXiv: https://arxiv.org/abs/2505.15510 · HTML: https://mlsanity.com/paper/2505.15510/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 45 of its 55 references that are in the map.
- First posted on arXiv 21 May 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 45)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models](https://mlsanity.com/paper/2406.09403/) (2024)
- [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023)
- [Imagine while Reasoning in Space: Multimodal Visualization-of-Thought](https://mlsanity.com/paper/2501.07542/) (2025)
- [Compositional Chain-of-Thought Prompting for Large Multimodal Models](https://mlsanity.com/paper/2311.17076/) (2023)
- [OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning](https://mlsanity.com/paper/2505.08617/) (2025)
- [DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models](https://mlsanity.com/paper/2310.16436/) (2023)
- [M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought](https://mlsanity.com/paper/2405.16473/) (2024)
- [ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation](https://mlsanity.com/paper/2407.06135/) (2024)
- [Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages](https://mlsanity.com/paper/2310.14799/) (2023)
- [Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models](https://mlsanity.com/paper/2312.03052/) (2023)
- [Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models](https://mlsanity.com/paper/2405.13872/) (2024)
- [T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering](https://mlsanity.com/paper/2305.03453/) (2023)

## Built on by (most cited first, 30 of 37)

- [Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers](https://mlsanity.com/paper/2506.23918/) (2025)
- [Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens](https://mlsanity.com/paper/2506.17218/) (2025)
- [Monet: Reasoning in Latent Visual Space Beyond Images and Language](https://mlsanity.com/paper/2511.21395/) (2025)
- [Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens](https://mlsanity.com/paper/2511.19418/) (2025)
- [AI4Research: A Survey of Artificial Intelligence for Scientific Research](https://mlsanity.com/paper/2507.01903/) (2025)
- [MathCanvas: Intrinsic Visual Chain-of-Thought for Multimodal Mathematical Reasoning](https://mlsanity.com/paper/2510.14958/) (2025)
- [Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning](https://mlsanity.com/paper/2510.12712/) (2025)
- [Code2World: A GUI World Model via Renderable Code Generation](https://mlsanity.com/paper/2602.09856/) (2026)
- [DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking](https://mlsanity.com/paper/2507.20879/) (2025)
- [OmniDrive-R1: Reinforcement-driven Interleaved Multi-modal Chain-of-Thought for Trustworthy Vision-Language Autonomous Driving](https://mlsanity.com/paper/2512.14044/) (2025)
- [InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search](https://mlsanity.com/paper/2512.18745/) (2025)
- [Deep But Reliable: Advancing Multi-turn Reasoning for Thinking with Images](https://mlsanity.com/paper/2512.17306/) (2025)
- [What, Whether and How? Unveiling Process Reward Models for Thinking with Images Reasoning](https://mlsanity.com/paper/2602.08346/) (2026)
- [PaLMR: Towards Faithful Visual Reasoning via Multimodal Process Alignment](https://mlsanity.com/paper/2603.06652/) (2026)
- [OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model](https://mlsanity.com/paper/2604.20806/) (2026)
- [See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles](https://mlsanity.com/paper/2509.13615/) (2025)
- [Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agents](https://mlsanity.com/paper/2510.02204/) (2025)
- [Locate-Then-Examine: Grounded Region Reasoning Improves Detection of AI-Generated Images](https://mlsanity.com/paper/2510.04225/) (2025)
- [Thinking with Images via Self-Calling Agent](https://mlsanity.com/paper/2512.08511/) (2025)
- [Let's Think with Images Efficiently! An Interleaved-Modal Chain-of-Thought Reasoning Framework with Dynamic and Precise Visual Thoughts](https://mlsanity.com/paper/2603.21754/) (2026)
- [FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-Language Navigation](https://mlsanity.com/paper/2601.13976/) (2026)
- [Generative Visual Chain-of-Thought for Image Editing](https://mlsanity.com/paper/2603.01893/) (2026)
- [Self-Consistent Latent Reasoning: Long Latent Sequence Reasoning for Vision-Language Model](https://mlsanity.com/paper/2605.12163/) (2026)
- [AutoPR: Let's Automate Your Academic Promotion!](https://mlsanity.com/paper/2510.09558/) (2025)
- [Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Tasks](https://mlsanity.com/paper/2512.01725/) (2025)
- [ForgeryVCR: Visual-Centric Reasoning via Efficient Forensic Tools in MLLMs for Image Forgery Detection and Localization](https://mlsanity.com/paper/2602.14098/) (2026)
- [Measuring AI Reasoning: A Guide for Researchers](https://mlsanity.com/paper/2605.02442/) (2026)
- [When Think-with-Image Meets Safety: What Determines Multimodal Jailbreak Robustness?](https://mlsanity.com/paper/2605.27932/) (2026)
- [HoloGeo: Mitigating Landmark Bias in Geo-localization via Evidence-Driven Reasoning](https://mlsanity.com/paper/2607.15255/) (2026)
- [Thinking with Comics: Enhancing Multimodal Reasoning through Structured Visual Storytelling](https://mlsanity.com/paper/2602.02453/) (2026)

## Abstract (arXiv)

Large Vision-Language Models (LVLMs) have achieved significant success in multimodal tasks, with multimodal chain-of-thought (MCoT) further enhancing performance and interpretability. Recent MCoT methods fall into two categories: (i) Textual-MCoT (T-MCoT), which takes multimodal input and produces textual output; and (ii) Interleaved-MCoT (I-MCoT), which generates interleaved image-text outputs. Despite advances in both approaches, the mechanisms driving these improvements are not fully understood. To fill this gap, we first reveal that MCoT boosts LVLMs by incorporating visual thoughts, which convey image information to the reasoning process regardless of the MCoT format, depending only on clarity and conciseness of expression. Furthermore, to explore visual thoughts systematically, we define four distinct forms of visual thought expressions and analyze them comprehensively. Our findings demonstrate that these forms differ in clarity and conciseness, yielding varying levels of MCoT improvement. Additionally, we explore the internal nature of visual thoughts, finding that visual thoughts serve as intermediaries between the input image and reasoning to deeper transformer layers, enabling more advanced visual information transmission. We hope that the visual thoughts can inspire further breakthroughs for future MCoT research.