mlsanityData to 7 Oct 2026

Reinforcing VLMs to Use Tools for Detailed Visual Reasoning Under Resource Constraints

arXiv 2506.14821 ↗First posted 10 Jun 2025cs.LG cs.AI cs.CVcited by 6

On the map: LLM reasoning → Multimodal reasoning

also in 2 more
Abstract (arXiv)

Despite tremendous recent advances in large model reasoning ability, vision-language models (VLMs) still struggle with detailed visual reasoning, especially when compute resources are limited. To address this challenge, we draw inspiration from methods like Deepseek-r1 for VLMs and train smaller-scale models with Group Relative Policy Optimization (GRPO) to use external tools such as zoom. The greatest benefit is obtained with a combination of GRPO learning, a simple reward structure, a simplified tool-calling interface, allocating additional tokens to the result of the tool call, and a training data mix that over-represents visually difficult examples. Compared to similarly-sized baseline models, our method achieves better performance on some visual question-answering (VQA) tasks, thanks to the detailed visual information gathered from the external tool.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 38 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

26 of 38 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning20 May
  2. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning10 Apr
  3. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr
  4. 2025OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles21 Mar
  5. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization13 Mar
  6. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models9 Mar
  7. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding15 Apr
  8. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action20 Mar
  9. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models8 Mar

Other references · 17

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2023Qwen Technical Reportcited by 3,273
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  8. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  9. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  10. 2024ZoomEye: Enhancing Multimodal LLMs with Human-Like Zooming Capabilities through Tree-Based Image Explorationcited by 99
  11. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 86
  12. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learningcited by 68
5 more
  1. 2025DyFo: A Training-Free Dynamic Focus Visual Search for Enhancing LMMs in Fine-Grained Visual Understandingcited by 47
  2. 2024InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understandingcited by 11
  3. 2024How Well Can Vision Language Models See Image Details?cited by 10
  4. 2024Dragonfly: Multi-Resolution Zoom-In Encoding Enhances Vision-Language Modelscited by 8
  5. 2024LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformercited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2026SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMscited by 6
  2. 2026Visual Reasoning through Tool-supervised Reinforcement Learningcited by 4
  3. 2025Zooming into Comics: Region-Aware RL Improves Fine-Grained Comic Understanding in Vision-Language Modelscited by 1
  4. 2026Beyond Zooming: Learning Multi-Tool Visual Reasoning for Ultra-High-Resolution Remote Sensingcited by 1
  5. 2026Robusto-2: Benchmarking Humans & VLMs for Autonomous Driving in Lima & New York Citycited by 0
  6. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Design
  2. 2024Probing Visual Language Priors in VLMs
  3. 2025Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning
  4. 2025UAV-VL-R1: Generalizing Vision-Language Models via Supervised Fine-Tuning and Multi-Stage GRPO for UAV Visual Reasoning
  5. 2024From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesis
  6. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
  7. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models
  8. 2025When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs