Perceive, Interact, Reason: Building Tool-Augmented Visual Agents for Spatial Reasoning
On the map: LLM reasoning → Spatial and embodied reasoning
also in 2 more
Abstract (arXiv)
While recent vision-language models (VLMs) demonstrate strong multimodal understanding, they remain limited in spatial reasoning tasks that require active evidence acquisition and multi-step visual interaction. This limitation suggests that relying solely on implicit visual representations from vision encoders is insufficient for recovering fine-grained spatial evidence. We introduce PERception-Interaction-reason Agent (PERIA), a tool-augmented visual agent for spatial reasoning tasks across map reasoning, visual probing, and vision reconstruction. PERIA uses two lightweight tool families: vision perception tools for exposing textual, symbolic, and spatial evidence, and vision interaction tools for manipulating visual context, tracing paths, and verifying spatial relations. To train PERIA, we develop a unified recipe that combines supervised tool-use trajectory synthesis, composite rewards, and Observation-Relaxed Group-in-Group Policy Optimization (OR-GIGPO) for effective multi-tool behavior. Experiments on 13 benchmarks from 8 datasets show that PERIA-8B improves over the Qwen3-8B backbone by 10.0% on in-distribution benchmarks and 4.4% on out-of-distribution benchmarks, while outperforming previous state-of-the-art baselines of similar size by 7.0%-14.8%. It also achieves performance comparable to much larger models such as Qwen3-VL-235B-A22B-Thinking and GPT-5, demonstrating the effectiveness of PERIA in enhancing spatial reasoning capabilities.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 47 references that are in the map. First posted on arXiv 11 Jun 2026.
Builds on
43 of 47 references.
Closest ancestors · 6
Same direction, newest first
- 2026Think3D: Thinking with Space for Spatial Reasoning19 Jan
- 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models3 Jun
- 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models25 Mar
- 2024MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models31 Dec
- 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark10 Dec
- 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
Other references · 37
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
- 2025OpenAI GPT-5 System Cardcited by 989
25 more
- 2025SAM 3: Segment Anything with Conceptscited by 906
- 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
- 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
- 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
- 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 410
- 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
- 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 370
- 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
- 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 120
- 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96
- 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 90
- 2022MapQA: A Dataset for Question Answering on Choropleth Mapscited by 85
- 2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understandingcited by 83
- 2025PyVision: Agentic Vision with Dynamic Toolingcited by 79
- 2026PaddleOCR-VL-1.5: Towards a Multi-Task 0.9B VLM for Robust In-the-Wild Document Parsingcited by 57
- 2025VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learningcited by 51
- 2026BabyVision: Visual Reasoning Beyond Languagecited by 49
- 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 46
- 2025Can Large Vision Language Models Read Maps Like a Human?cited by 30
- 2025ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Mapscited by 23
- 2025RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learningcited by 21
- 2026Visual Generation Unlocks Human-Like Reasoning through Multimodal World Modelscited by 19
- 2025Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMscited by 17
- 2025VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenescited by 4
- 2025MapTrace: Scalable Data Generation for Route Tracing on Mapscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them
- 2026SpatiO: Adaptive Test-Time Orchestration of Vision-Language Agents for Spatial Reasoning
- 2026SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning
- 2026S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence
- 2026LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models
- 2026From Reasoning Failures to Composable Video Spatial Intelligence
- 2026SpotAgent: Grounding Visual Geo-localization in Large Vision-Language Models through Agentic Reasoning
- 2026Active Exploring like a Pigeon: Reinforcing Spatial Reasoning via Agentic Vision-Language Models