DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoning
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
The "thinking with images" paradigm represents a pivotal shift in the reasoning of Vision Language Models (VLMs), moving from text-dominant chain-of-thought to image-interactive reasoning. By invoking visual tools or generating intermediate visual representations, VLMs can iteratively attend to fine-grained regions, enabling deeper image understanding and more faithful multimodal reasoning. As an emerging paradigm, however, it still leaves substantial room for exploration in data construction accuracy, structural design, and broader application scenarios, which offer rich opportunities for advancing multimodal reasoning. To further advance this line of work, we present DeepSketcher, a comprehensive suite comprising both an image-text interleaved dataset and a self-contained model. The dataset contains 31k chain-of-thought (CoT) reasoning trajectories with diverse tool calls and resulting edited images, covering a wide range of data types and manipulation instructions with high annotation accuracy. Building on this resource, we design a model that performs interleaved image-text reasoning and natively generates "visual thoughts" by operating directly in the visual embedding space, rather than invoking external tools and repeatedly re-encoding generated images. This design enables tool-free and more flexible "thinking with images". Extensive experiments on multimodal reasoning benchmarks demonstrate strong performance, validating both the utility of the dataset and the effectiveness of the model design.
Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 45 of its 61 references that are in the map. First posted on arXiv 30 Sep 2025.
Builds on
45 of 61 references.
Closest ancestors · 24
Same direction, newest first
- 2025Self-Rewarding Vision-Language Model via Reasoning Decomposition27 Aug
- 2025Thyme: Think Beyond Images15 Aug
- 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning22 Jul
- 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens20 Jun
- 2025Metis-RISE: RL Incentivizes and SFT Enhances Multimodal Reasoning Model Learning16 Jun
- 2025ChartReasoner: Code-Driven Modality Bridging for Long-Chain Reasoning in Chart Question Answering11 Jun
- 2025MiMo-VL Technical Report4 Jun
- 2025Thinking with Generated Images28 May
- 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning20 May
- 2025MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning15 May
- 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning13 May
- 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models10 Apr
12 more
- 2025Mitigating Visual Forgetting via Take-along Visual Conditioning for Multi-modal Long CoT Reasoning17 Mar
- 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization13 Mar
- 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning10 Mar
- 2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding9 Jan
- 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts6 Jul
- 2024We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?1 Jul
- 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models13 Jun
- 2024MM-MATH: Advancing Multimodal Math Evaluation with Process Evaluation and Fine-grained Classification7 Apr
- 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset22 Feb
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
- 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar
- 2022Visual Programming: Compositional visual reasoning without training18 Nov
Other references · 21
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
- 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
- 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
- 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
- 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
- 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 228
- 2024Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Searchcited by 175
9 more
- 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 161
- 2022UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expressioncited by 137
- 2025Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawingcited by 136
- 2025OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoningcited by 69
- 2025Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generationcited by 58
- 2024List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMscited by 32
- 2024GeoGPT4V: Towards Geometric Multi-modal Large Language Models with Geometric Image Generationcited by 32
- 2025Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMscited by 23
- 2025Effective Training Data Synthesis for Improving MLLM Chart Understandingcited by 20
Built on by
Citing papers per year: 2025: 5; 2026: 9.
14 papers.
Sort:
By area:
- 2025V-Thinker: Interactive Thinking with Imagescited by 26
- 2025Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoningcited by 20
- 2025Mull-Tokens: Modality-Agnostic Latent Thinkingcited by 19
- 2026Imagination Helps Visual Reasoning, But Not Yet in Latent Spacecited by 12
- 2025Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestrationcited by 9
- 2026Beyond NL2Code: A Structured Survey of Multimodal Code Intelligencecited by 6
- 2026Improving Visual Reasoning with Iterative Evidence Refinementcited by 5
- 2026Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Taskscited by 4
- 2025Metis-HOME: Hybrid Optimized Mixture-of-Experts for Multimodal Reasoningcited by 2
- 2026Geo-Code: A Code Framework for Reverse Code Generation from Geometric Images Based on Two-Stage Multi-Agent Evolutioncited by 1
- 2026SketchVLM: Vision language models can annotate images to explain thoughts and guide userscited by 0
- 2026InterSketch: An Interleaved Reasoning Model with Self-correcting Visual Sketch and Stepwise Rewardcited by 0
2 more
- 2026Multimodal Thinking with Renderable Programs24 Sep 2026
- 2026Remember-R1: Mitigating Long-Context Visual Forgetting through Reinforcement Learning2 Aug 2026
- 2026Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence14 Jun 2026
- 2026InterSketch: An Interleaved Reasoning Model with Self-correcting Visual Sketch and Stepwise Reward26 May 2026
- 2026SketchVLM: Vision language models can annotate images to explain thoughts and guide users23 Apr 2026
- 2026Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks21 Apr 2026
- 2026Improving Visual Reasoning with Iterative Evidence Refinement14 Mar 2026
- 2026Imagination Helps Visual Reasoning, But Not Yet in Latent Space26 Feb 2026
- 2026Geo-Code: A Code Framework for Reverse Code Generation from Geometric Images Based on Two-Stage Multi-Agent Evolution8 Feb 2026
- 2025Mull-Tokens: Modality-Agnostic Latent Thinking11 Dec 2025
- 2025Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning23 Nov 2025
- 2025Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestration19 Nov 2025
2 more
- 2025V-Thinker: Interactive Thinking with Images6 Nov 2025
- 2025Metis-HOME: Hybrid Optimized Mixture-of-Experts for Multimodal Reasoning23 Oct 2025
- 2025V-Thinker: Interactive Thinking with Imagescited by 26
- 2025Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoningcited by 20
- 2026Imagination Helps Visual Reasoning, But Not Yet in Latent Spacecited by 12
- 2025Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestrationcited by 9
- 2026Improving Visual Reasoning with Iterative Evidence Refinementcited by 5
- 2026Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Taskscited by 4
- 2025Metis-HOME: Hybrid Optimized Mixture-of-Experts for Multimodal Reasoningcited by 2
- 2026Geo-Code: A Code Framework for Reverse Code Generation from Geometric Images Based on Two-Stage Multi-Agent Evolutioncited by 1
- 2026SketchVLM: Vision language models can annotate images to explain thoughts and guide userscited by 0
- 2026InterSketch: An Interleaved Reasoning Model with Self-correcting Visual Sketch and Stepwise Rewardcited by 0
- 2026Remember-R1: Mitigating Long-Context Visual Forgetting through Reinforcement Learningcited by 0
- 2026Multimodal Thinking with Renderable Programscited by 0
- 2026Multimodal Thinking with Renderable Programs24 Sep 2026
- 2026Remember-R1: Mitigating Long-Context Visual Forgetting through Reinforcement Learning2 Aug 2026
- 2026InterSketch: An Interleaved Reasoning Model with Self-correcting Visual Sketch and Stepwise Reward26 May 2026
- 2026SketchVLM: Vision language models can annotate images to explain thoughts and guide users23 Apr 2026
- 2026Unveiling Fine-Grained Visual Traces: Evaluating Multimodal Interleaved Reasoning Chains in Multimodal STEM Tasks21 Apr 2026
- 2026Improving Visual Reasoning with Iterative Evidence Refinement14 Mar 2026
- 2026Imagination Helps Visual Reasoning, But Not Yet in Latent Space26 Feb 2026
- 2026Geo-Code: A Code Framework for Reverse Code Generation from Geometric Images Based on Two-Stage Multi-Agent Evolution8 Feb 2026
- 2025Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning23 Nov 2025
- 2025Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestration19 Nov 2025
- 2025V-Thinker: Interactive Thinking with Images6 Nov 2025
- 2025Metis-HOME: Hybrid Optimized Mixture-of-Experts for Multimodal Reasoning23 Oct 2025
- 2025Mull-Tokens: Modality-Agnostic Latent Thinkingcited by 19
- 2026Beyond NL2Code: A Structured Survey of Multimodal Code Intelligencecited by 6
- 2026Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence14 Jun 2026
- 2025Mull-Tokens: Modality-Agnostic Latent Thinking11 Dec 2025
Similar, not cited
Close in wording, no citation either way
- 2025Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning
- 2025Thinking with Programming Vision: Towards a Unified View for Thinking with Images
- 2025Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward
- 2026Reasoning with Image Generation
- 2025VGR: Visual Grounded Reasoning
- 2025SIFThinker: Spatially-Aware Image Focus for Visual Reasoning
- 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
- 2026Distilling Visual Reasoning into Text Space