InterSketch: An Interleaved Reasoning Model with Self-correcting Visual Sketch and Stepwise Reward
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
While vision-language models (VLMs) have exhibited multi-turn visual reasoning capabilities, their reasoning trajectories remain relatively shallow and are dominated by a text-centric paradigm, limiting their applicability to complex visual challenges. In contrast, human-like thought typically involves long-horizon reasoning with an interleaved visual-textual chain-of-thought (VT-CoT). To bridge this gap, we introduce InterSketch, an interleaved reasoning model to enhance the VT-CoT capability via self-correcting and stepwise reward mechanisms. InterSketch dynamically generates intermediate visual sketches using external tools and interleaves them with textual reasoning, enabling effective perception and logical reasoning over long-horizon visual understanding tasks. Specifically, in the first cold-start stage, we propose a synthesized high-quality interleaved VT-CoT dataset and include a reflection mechanism to enable the model's capability in multi-turn interleaved reasoning and self-correction. In the subsequent reinforcement learning (RL) stage, we design a stepwise reward mechanism to mitigate the sparsity of reward signals inherent in end-only supervision over long-horizon reasoning. Extensive experiments on visual reasoning benchmarks demonstrate the effectiveness of InterSketch, even outperforming proprietary models such as Gemini-3-Pro.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 35 references that are in the map. First posted on arXiv 26 May 2026.
Builds on
34 of 35 references.
Closest ancestors · 17
Same direction, newest first
- 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learning30 Dec
- 2025Thinking with Programming Vision: Towards a Unified View for Thinking with Images3 Dec
- 2025TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning3 Nov
- 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoning30 Oct
- 2025DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoning30 Sep
- 2025Reinforced Visual Perception with Tools1 Sep
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency25 Aug
- 2025Thyme: Think Beyond Images15 Aug
- 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers30 Jun
- 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use25 May
- 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning21 May
- 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning20 May
5 more
- 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning13 May
- 2025Imagine while Reasoning in Space: Multimodal Visualization-of-Thought13 Jan
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
- 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
- 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar
Other references · 17
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2024GPT-4o System Cardcited by 4,467
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
- 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
- 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
- 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
- 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
5 more
- 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
- 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 116
- 2024VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMscited by 47
- 2025RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmarkcited by 33
- 2025Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmarkcited by 28
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning
- 2025Self-Rewarding Vision-Language Model via Reasoning Decomposition
- 2026Render-of-Thought: Rendering Textual Chain-of-Thought as Images for Visual Latent Reasoning
- 2025Interleaved Latent Visual Reasoning with Selective Perceptual Modeling
- 2025Thinking with Images via Self-Calling Agent
- 2026See Further, Think Deeper: Advancing VLM's Reasoning Ability with Low-level Visual Cues and Reflection
- 2025Unveiling Chain of Step Reasoning for Vision-Language Models with Fine-grained Rewards
- 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models