What Does Vision Tool-Use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoom
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
Vision tool-use reinforcement learning (RL) can equip vision language models with visual operators such as crop-and-zoom and achieves strong performance gains, yet it remains unclear whether these gains are driven by improvements in tool use or evolving intrinsic capabilities. We introduce MED (Measure--Explain--Diagnose), a coarse-to-fine framework that disentangles intrinsic capability changes from tool-induced effects, decomposes the tool-induced performance difference into gain and harm terms, and probes the mechanisms driving their evolution. Across checkpoint-level analyses in the crop-and-zoom setting on two VLMs with different tool priors and six benchmarks, we find that improvements are dominated by intrinsic learning, while tool-use RL mainly reduces tool-induced harm (e.g., fewer call-induced errors and weaker tool schema interference) and yields limited progress in tool-based correction of intrinsic failures. Overall, in the crop-and-zoom setting studied here, current vision tool-use RL learns to coexist safely with tools rather than master them.
Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 46 references that are in the map. First posted on arXiv 1 Feb 2026.
Builds on
36 of 46 references.
Closest ancestors · 5
Same direction, newest first
- 2025AdaTooler-V: Adaptive Tool-Use for Images and Videos18 Dec
- 2025Knowing the Answer Isn't Enough: Fixing Reasoning Path Failures in LVLMs6 Dec
- 2025Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs21 May
- 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning20 May
- 2025OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning13 May
Other references · 31
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
- 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
- 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
- 2025Video-R1: Reinforcing Video Reasoning in MLLMscited by 464
- 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
- 2019Scene Text Visual Question Answeringcited by 327
19 more
- 2019PlotQA: Reasoning over Scientific Plotscited by 317
- 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
- 2015Evaluation of Deep Convolutional Nets for Document Image Classification and Retrievalcited by 212
- 2025Thyme: Think Beyond Imagescited by 160
- 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Modelscited by 151
- 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
- 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 116
- 2025VideoRFT: Incentivizing Video Reasoning Capability in MLLMs via Reinforced Fine-Tuningcited by 86
- 2020On the General Value of Evidence, and Bilingual Scene-Text Visual Question Answeringcited by 78
- 2024SPIQA: A Dataset for Multimodal Question Answering on Scientific Paperscited by 75
- 2025ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learningcited by 41
- 2025Reinforced Visual Perception with Toolscited by 38
- 2025MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMscited by 31
- 2025Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearchcited by 25
- 2025HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decouplingcited by 16
- 2025From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoningcited by 9
- 2025Revisiting the Necessity of Lengthy Chain-of-Thought in Vision-centric Reasoning Generalizationcited by 8
- 2025VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?cited by 7
- 2022Hamming Distributions of Popular Perceptual Hashing Techniquescited by 4
Built on by
Citing papers per year: 2026: 9.
9 papers.
- 2026The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Imagescited by 2
- 2026Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoningcited by 2
- 2026Beyond Zooming: Learning Multi-Tool Visual Reasoning for Ultra-High-Resolution Remote Sensingcited by 1
- 2026Thinking Once Is Enough: Intermediate-Layer Evidence Routing for High-Resolution VQAcited by 1
- 2026Beacon: Knowing When and How to Perform Agentic Visual Reasoningcited by 1
- 2026VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenescited by 0
- 2026Region-Level Policy Optimization for Fine-grained MLLM Perceptioncited by 0
- 2026When Should a VLM Look? Paying Only for Visual Calls That Were Needed and Usedcited by 0
- 2026Prompt, Probe, Train, or Annotate? Single-camera sports video understanding in amateur settingscited by 0
- 2026Prompt, Probe, Train, or Annotate? Single-camera sports video understanding in amateur settings23 Sep 2026
- 2026When Should a VLM Look? Paying Only for Visual Calls That Were Needed and Used19 Sep 2026
- 2026Region-Level Policy Optimization for Fine-grained MLLM Perception17 Sep 2026
- 2026Thinking With Tools, Not With Pixels: Tool Calls as Text Scaffolds for Visual Reasoning10 Aug 2026
- 2026The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images6 Aug 2026
- 2026Beacon: Knowing When and How to Perform Agentic Visual Reasoning30 Jul 2026
- 2026Thinking Once Is Enough: Intermediate-Layer Evidence Routing for High-Resolution VQA30 Jul 2026
- 2026Beyond Zooming: Learning Multi-Tool Visual Reasoning for Ultra-High-Resolution Remote Sensing28 Jul 2026
- 2026VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenes25 May 2026
Similar, not cited
Close in wording, no citation either way
- 2025VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selection
- 2026ParaVT: Taming the Tool Prior Paradox for Parallel Tool Use in Agentic Video Reinforcement Learning
- 2026VC-Tooler: Learning Compositional and Adaptive Visual Tool Use
- 2026ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents
- 2026GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents
- 2026Visual Reasoning through Tool-supervised Reinforcement Learning
- 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
- 2025Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme