mlsanityData to 14 Oct 2026

VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Use

arXiv 2505.19255 ↗First posted 25 May 2025cs.LG cs.AIcited by 90

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Reinforcement Learning Finetuning (RFT) has significantly advanced the reasoning capabilities of large language models (LLMs) by enabling long chains of thought, self-correction, and effective tool use. While recent works attempt to extend RFT to vision-language models (VLMs), these efforts largely produce text-only reasoning conditioned on static image inputs, falling short of true multimodal reasoning in the response. In contrast, test-time methods like Visual Sketchpad incorporate visual steps but lack training mechanisms. We introduce VTool-R1, the first framework that trains VLMs to generate multimodal chains of thought by interleaving text and intermediate visual reasoning steps. VTool-R1 integrates Python-based visual editing tools into the RFT process, enabling VLMs to learn when and how to generate visual reasoning steps that benefit final reasoning. Trained with outcome-based rewards tied to task accuracy, our approach elicits strategic visual tool use for reasoning without relying on process-based supervision. Experiments on structured visual question answering over charts and tables show that VTool-R1 enhances reasoning performance by teaching VLMs to "think with images" and generate multimodal chain of thoughts with tools. To support future research in multi-turn multi-modal reasoning, we open-source our code at https://github.com/VTOOL-R1/vtool-r1

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 44 references that are in the map. First posted on arXiv 25 May 2025.

Builds on

33 of 44 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning10 Apr
  2. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models9 Mar
  3. 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement9 Mar
  4. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Model7 Mar
  5. 2025Cache-of-Thought: Master-Apprentice Framework for Cost-Effective Vision Language Model Reasoning27 Feb
  6. 2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding9 Jan
  7. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Step15 Nov
  8. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models13 Jun
  9. 2023ViperGPT: Visual Inference via Python Execution for Reasoning14 Mar
  10. 2022DePlot: One-shot visual language reasoning by plot-to-table translation20 Dec
  11. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering19 Dec
  12. 2022Visual Programming: Compositional visual reasoning without training18 Nov
1 more
  1. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning19 Mar

Other references · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2025Qwen2.5-VL Technical Reportcited by 5,587
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  11. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  12. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
8 more
  1. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  2. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  3. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 392
  4. 2025R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimizationcited by 301
  5. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  6. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 161
  7. 2024TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domainscited by 69
  8. 2024Can Large Vision-Language Models Correct Semantic Grounding Errors By Themselves?cited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 27; 2026: 63.

90 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  3. 2025MemGen: Weaving Generative Latent Memory for Self-Evolving Agentscited by 109
  4. 2025Latent Visual Reasoningcited by 104
  5. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 65
  6. 2025StreamAgent: Towards Anticipatory Agents for Streaming Video Understandingcited by 33
  7. 2025Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedbackcited by 32
  8. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 28
  9. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Modelscited by 26
  10. 2025V-Thinker: Interactive Thinking with Imagescited by 26
  11. 2025SIFThinker: Spatially-Aware Image Focus for Visual Reasoningcited by 25
  12. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 24
38 more
  1. 2026Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddingscited by 24
  2. 2025SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RLcited by 21
  3. 2025Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMscited by 17
  4. 2026LensWalk: Agentic Video Understanding by Planning How You See in Videoscited by 17
  5. 2026LaViT: Aligning Latent Visual Thoughts for Multi-modal Reasoningcited by 15
  6. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 14
  7. 2026Explore with Long-term Memory: A Benchmark and Multimodal LLM-based Reinforcement Learning Framework for Embodied Explorationcited by 13
  8. 2026Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Modelscited by 10
  9. 2025Yo'City: Personalized and Boundless 3D Realistic City Scene Generation via Self-Critic Expansioncited by 9
  10. 2025Enhancing Agentic RL with Progressive Reward Shaping and Value-based Sampling Policy Optimizationcited by 9
  11. 2026Thinking Without Images: Internalizing Visual Manipulation with On-Policy Self-Distillationcited by 9
  12. 2025Aha Moment Revisited: Are VLMs Truly Capable of Self Verification in Inference-time Scaling?cited by 8
  13. 2025Position: Reasoning After Perception Means Reasoning Without Visioncited by 8
  14. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  15. 2025VG-Refiner: Towards Tool-Refined Referring Grounded Reasoning via Agentic Reinforcement Learningcited by 8
  16. 2026MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learningcited by 8
  17. 2026UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMscited by 8
  18. 2025Mixture-of-Minds: Multi-Agent Reinforcement Learning for Table Understandingcited by 7
  19. 2026Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policycited by 6
  20. 2026Leveraging Latent Visual Reasoning in Silencecited by 5
  21. 2025CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questionscited by 4
  22. 2025ReVSeg: Incentivizing the Reasoning Chain for Video Segmentation with Reinforcement Learningcited by 4
  23. 2025Training Multi-Image Vision Agents via End2End Reinforcement Learningcited by 4
  24. 2025CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal Reasoningcited by 4
  25. 2026Visual Reasoning through Tool-supervised Reinforcement Learningcited by 4
  26. 2026Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gainscited by 4
  27. 2025See Less, See Right: Bi-directional Perceptual Shaping For Multimodal Reasoningcited by 3
  28. 2026StruVis: Enhancing Reasoning-based Text-to-Image Generation via Thinking with Structured Visioncited by 3
  29. 2026Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoningcited by 3
  30. 2026Audio-Visual Intelligence in Large Foundation Modelscited by 2
  31. 2026LensVLM: Selective Context Expansion for Compressed Visual Representation of Textcited by 2
  32. 2026Dynamo: Dynamic Skill-Tool Evolution for Vision-Language Agentscited by 2
  33. 2026PixelEyes: Decoupling Perception and Reasoning for Pinpoint Visual Evidence Seekingcited by 2
  34. 2026OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documentscited by 2
  35. 2026SPyCE: Skill-Policy Co-evolution for Multimodal Agentscited by 2
  36. 2025Augmented Vision-Language Models: A Systematic Reviewcited by 1
  37. 2026V-Reflection: Transforming MLLMs from Passive Observers to Active Interrogatorscited by 1
  38. 2026Reversing the Flow: Generation-to-Understanding Synergy in Large Multimodal Modelscited by 1

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026See Further, Think Deeper: Advancing VLM's Reasoning Ability with Low-level Visual Cues and Reflection
  2. 2026Genesis: Visual Augmentation Primitives for Multimodal Chain-of-Thought Reasoning
  3. 2025Activating Visual Context and Commonsense Reasoning through Masked Prediction in VLMs
  4. 2025Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning
  5. 2026See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs
  6. 2025Point-RFT: Improving Multimodal Reasoning with Visually Grounded Reinforcement Finetuning
  7. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models
  8. 2025Perception Before Reasoning: Two-Stage Reinforcement Learning for Visual Reasoning in Vision-Language Models