mlsanityData to 14 Oct 2026

When Should a VLM Look? Paying Only for Visual Calls That Were Needed and Used

arXiv 2609.22910 ↗First posted 19 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Vision-language agents that crop and zoom are trained with rewards that credit a successful tool call, yet a successful call does not show that the model needed to look or used the pixels it received. On our cold-start checkpoint only 10% to 12% of visual calls were both needed and used, and released agents make spurious calls 36% to 87% of the time on individual benchmarks. Outcome rewards, judge rewards, and branch probes each observe one side of this failure, and about two thirds of what an outcome reward pays goes to calls that were neither needed nor used. CounterCredit asks both questions of every image-returning call at its realized pre-call state, using the policy's own gold-answer score. A decision value compares the realized visual branch with answering immediately; an evidence value compares the returned crop with random same-size patches substituted into the same call. A call verified on both earns cashback and every other executed call pays rent; the price is bounded so that every correct trajectory outranks every wrong one, and a dual-channel GRPO advantage keeps the price in its own units. From the same cold start, prompt pool, and budget, CounterCredit reaches 89.5% on V*, 80.2% on HR-Bench-4K, and 76.4% on HR-Bench-8K, 6.3 to 9.4 points above outcome-only GRPO at 1.78 against 1.84 calls per question, and lowers the spurious-call rate to 31% to 36%, the lowest among the agents evaluated. The same recipe lifts a Qwen3-VL-8B base from 75.4 to 80.8 on average.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 49 references that are in the map. First posted on arXiv 19 Sep 2026.

Builds on

42 of 49 references.

References · 42

  1. 2025Qwen2.5-VL Technical Reportcited by 5,587
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  4. 2025Qwen3-VL Technical Reportcited by 2,793
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  6. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  7. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  8. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  9. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  10. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  11. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  12. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
30 more
  1. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 410
  2. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  3. 2013Direct and Indirect Effectscited by 280
  4. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 278
  5. 2025Thyme: Think Beyond Imagescited by 167
  6. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
  7. 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 120
  8. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 103
  9. 2025Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMscited by 96
  10. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 90
  11. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 48
  12. 2025ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learningcited by 43
  13. 2025CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimizationcited by 26
  14. 2025AdaTooler-V: Adaptive Tool-Use for Images and Videoscited by 21
  15. 2025On the Faithfulness of Visual Thinking: Measurement and Enhancementcited by 17
  16. 2026PyVision-RL: Forging Open Agentic Vision Models via RLcited by 16
  17. 2025TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixturecited by 15
  18. 2025PACR: Progressively Ascending Confidence Reward for LLM Reasoningcited by 9
  19. 2025From Illusion to Intention: Visual Rationale Learning for Vision-Language Reasoningcited by 9
  20. 2026What Does Vision Tool-Use Reinforcement Learning Really Learn? Disentangling Tool-Induced and Intrinsic Effects for Crop-and-Zoomcited by 9
  21. 2026Staying VIGILant: Mitigating Visual Laziness via Counterfactual Visual Alignment in MLLMscited by 8
  22. 2026Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoningcited by 7
  23. 2024MageBench: Bridging Large Multimodal Models to Agentscited by 6
  24. 2026TACO: Tool-Augmented Credit Optimization for Agentic Tool Usecited by 5
  25. 2026Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gainscited by 4
  26. 2026The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Imagescited by 2
  27. 2026The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidencecited by 2
  28. 2026Learning to Focus and Precise Cropping: A Reinforcement Learning Framework with Information Gaps and Grounding Loss for MLLMscited by 1
  29. 2026FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verificationcited by 1
  30. 2026Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026When the Judge Acts: Auditing VLM-Guided Image Selection on Culturally Situated Prompts
  2. 2026Why VLMs Miss Small Objects, and When Zooming In Is Safe
  3. 2025Visually Prompted Benchmarks Are Surprisingly Fragile
  4. 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agents
  5. 2026How Far Can VLMs Go for Visual Bug Detection? Studying 19,738 Keyframes from 41 Hours of Gameplay Videos
  6. 2026Same Reward, Different Skills: When Multimodal RL Learns to Look
  7. 2026Does a Learned Corrector Beat a Simple Retreat? Evidence from a Frozen VLA
  8. 2026Visual Credit Audit for Multimodal Spatial Reasoning