# VC-Tooler: Learning Compositional and Adaptive Visual Tool Use

arXiv: https://arxiv.org/abs/2608.02217 · HTML: https://mlsanity.com/paper/2608.02217/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 35 of its 46 references that are in the map.
- First posted on arXiv 3 Aug 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 35)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [LLaVA-CoT: Let Vision Language Models Reason Step-by-Step](https://mlsanity.com/paper/2411.10440/) (2024)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization](https://mlsanity.com/paper/2503.10615/) (2025)
- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?](https://mlsanity.com/paper/2408.13257/) (2024)
- [Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning](https://mlsanity.com/paper/2505.15966/) (2025)
- [Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers](https://mlsanity.com/paper/2506.23918/) (2025)
- [Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search](https://mlsanity.com/paper/2509.07969/) (2025)
- [WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent](https://mlsanity.com/paper/2508.05748/) (2025)
- [Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning](https://mlsanity.com/paper/2507.16746/) (2025)
- [Monet: Reasoning in Latent Visual Space Beyond Images and Language](https://mlsanity.com/paper/2511.21395/) (2025)
- [PyVision: Agentic Vision with Dynamic Tooling](https://mlsanity.com/paper/2507.07998/) (2025)
- [Thinking with Generated Images](https://mlsanity.com/paper/2505.22525/) (2025)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning](https://mlsanity.com/paper/2511.01833/) (2025)
- [Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning](https://mlsanity.com/paper/2507.05255/) (2025)
- [Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs](https://mlsanity.com/paper/2511.19773/) (2025)
- [CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoning](https://mlsanity.com/paper/2512.17312/) (2025)
- [PyVision-RL: Forging Open Agentic Vision Models via RL](https://mlsanity.com/paper/2602.20739/) (2026)
- [VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining](https://mlsanity.com/paper/2603.15030/) (2026)
- [ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch](https://mlsanity.com/paper/2601.13606/) (2026)
- [Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs](https://mlsanity.com/paper/2603.02556/) (2026)

## Abstract (arXiv)

Agentic multimodal reasoning extends passive image understanding by allowing VLMs to actively acquire and refine visual evidence through visual tool interactions. Effective visual tool use requires three capabilities: grounding tool calls in visual context, composing tools across multiple steps, and adapting reasoning to tool-returned observations. However, existing approaches largely focus on grounding within fixed tool spaces and rigid invocation patterns, leaving composition and adaptation insufficiently addressed. We present VC-Tooler, which learns visual tool use as a compositional and adaptive capability. To this end, we first build a trajectory bank through a hierarchical synthesis pipeline covering three capability levels: single-tool grounding, multi-tool composition, and diverse tool contexts and interfaces. We then train the model in two stages: a supervised cold start that establishes these capabilities, followed by reinforcement learning that encourages accurate, efficient, and context-aware visual tool use. VC-Tooler achieves state-of-the-art performance among open-source models on both general-purpose and agentic benchmarks, including 95.8% on V* and 35.3% on VTC-Bench, and shows promising transfer under richer tool settings at inference time. Project page: https://w1zheng.github.io/VC-Tooler