# Don't Show Pixels, Show Cues: Unlocking Visual Tool Reasoning in Language Models via Perception Programs

arXiv: https://arxiv.org/abs/2604.12896 · HTML: https://mlsanity.com/paper/2604.12896/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 14 Apr 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 17 of 17)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [Thyme: Think Beyond Images](https://mlsanity.com/paper/2508.11630/) (2025)
- [GRIT: Teaching MLLMs to Think with Images](https://mlsanity.com/paper/2505.15879/) (2025)
- [Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokens](https://mlsanity.com/paper/2506.17218/) (2025)
- [PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding](https://mlsanity.com/paper/2504.13180/) (2025)
- [Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning](https://mlsanity.com/paper/2507.16746/) (2025)
- [Grounded Reinforcement Learning for Visual Reasoning](https://mlsanity.com/paper/2505.23678/) (2025)
- [Reinforced Visual Perception with Tools](https://mlsanity.com/paper/2509.01656/) (2025)
- [Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning](https://mlsanity.com/paper/2507.05255/) (2025)
- [Introducing Visual Perception Token into Multimodal Large Language Model](https://mlsanity.com/paper/2502.17425/) (2025)
- [VipAct: Visual-Perception Enhancement via Specialized VLM Agent Collaboration and Tool-use](https://mlsanity.com/paper/2410.16400/) (2024)
- [Hidden in plain sight: VLMs overlook their visual representations](https://mlsanity.com/paper/2506.08008/) (2025)
- [Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learning](https://mlsanity.com/paper/2507.00748/) (2025)
- [MMFactory: A Universal Solution Search Engine for Vision-Language Tasks](https://mlsanity.com/paper/2412.18072/) (2024)

## Built on by (most cited first, 4 of 4)

- [Aphanta: Diagnosing Task-Aligned Image-Edited Intermediates for Multimodal Reasoning](https://mlsanity.com/paper/2608.26993/) (2026)
- [Seek-and-View Reasoning for Multi-View Spatial Understanding](https://mlsanity.com/paper/2610.11810/) (2026)
- [VisualCoding: Learning Executable Skills for Reliable Visual Reasoning](https://mlsanity.com/paper/2610.14403/) (2026)
- [Write Once, Plan Many: Building Reusable Visual Planning Systems from One Scene](https://mlsanity.com/paper/2610.19234/) (2026)

## Abstract (arXiv)

Multimodal language models (MLLMs) are increasingly paired with vision tools (e.g., depth, flow, correspondence) to enhance visual reasoning. However, despite access to these tool-generated visual cues, MLLMs often fail to benefit from them. Existing approaches typically feed raw tool outputs into the model, but these dense, pixel-level representations are misaligned with the language-native reasoning strengths of LLMs, leading to weak perception and reliance on language priors. We argue that, in problems where vision tools can provide the necessary visual cues, the bottleneck is not more tool calls or larger MLLMs, it is how tool outputs are represented. We introduce Perception Programs (P2), a training-free, model-agnostic method that rewrites tool outputs into compact, structured, language-native summaries that MLLMs can directly parse and reason over. Across six perception-centric tasks in BLINK, P2 consistently yields large improvements over base models and raw tool-augmented baselines. With GPT-5 Mini as the base model, P2 raises its accuracy from 41.35% to 86.47% on multi-view reasoning, from 52.42% to 81.45% on relative depth, and achieves a 22% average gain across tasks, setting new state-of-the-art results. Even on smaller MLLMs, e.g., InternVL3.5-4B and Qwen3VL-4B, we observe 15-40% absolute gains from P2, surpassing prior agentic, supervised, and RL-based tool-use methods-without any training or model modifications.