mlsanityData to 14 Oct 2026

Perceive, Interact, Reason: Building Tool-Augmented Visual Agents for Spatial Reasoning

arXiv 2606.12830 ↗First posted 11 Jun 2026cs.CV cs.AIcited by 0

On the map: LLM reasoning → Spatial and embodied reasoning

also in 2 more
Abstract (arXiv)

While recent vision-language models (VLMs) demonstrate strong multimodal understanding, they remain limited in spatial reasoning tasks that require active evidence acquisition and multi-step visual interaction. This limitation suggests that relying solely on implicit visual representations from vision encoders is insufficient for recovering fine-grained spatial evidence. We introduce PERception-Interaction-reason Agent (PERIA), a tool-augmented visual agent for spatial reasoning tasks across map reasoning, visual probing, and vision reconstruction. PERIA uses two lightweight tool families: vision perception tools for exposing textual, symbolic, and spatial evidence, and vision interaction tools for manipulating visual context, tracing paths, and verifying spatial relations. To train PERIA, we develop a unified recipe that combines supervised tool-use trajectory synthesis, composite rewards, and Observation-Relaxed Group-in-Group Policy Optimization (OR-GIGPO) for effective multi-tool behavior. Experiments on 13 benchmarks from 8 datasets show that PERIA-8B improves over the Qwen3-8B backbone by 10.0% on in-distribution benchmarks and 4.4% on out-of-distribution benchmarks, while outperforming previous state-of-the-art baselines of similar size by 7.0%-14.8%. It also achieves performance comparable to much larger models such as Qwen3-VL-235B-A22B-Thinking and GPT-5, demonstrating the effectiveness of PERIA in enhancing spatial reasoning capabilities.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 47 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

43 of 47 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Think3D: Thinking with Space for Spatial Reasoning19 Jan
  2. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models3 Jun
  3. 2025Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models25 Mar
  4. 2024MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models31 Dec
  5. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmark10 Dec
  6. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan

Other references · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  7. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2025Qwen3-VL Technical Reportcited by 2,793
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  12. 2025OpenAI GPT-5 System Cardcited by 989
25 more
  1. 2025SAM 3: Segment Anything with Conceptscited by 906
  2. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  3. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  4. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
  5. 2025DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learningcited by 410
  6. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  7. 2025R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalizationcited by 370
  8. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 125
  9. 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 120
  10. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96
  11. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 90
  12. 2022MapQA: A Dataset for Question Answering on Choropleth Mapscited by 85
  13. 2025ReFocus: Visual Editing as a Chain of Thought for Structured Image Understandingcited by 83
  14. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 79
  15. 2026PaddleOCR-VL-1.5: Towards a Multi-Task 0.9B VLM for Robust In-the-Wild Document Parsingcited by 57
  16. 2025VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learningcited by 51
  17. 2026BabyVision: Visual Reasoning Beyond Languagecited by 49
  18. 2024ChartMoE: Mixture of Diversely Aligned Expert Connector for Chart Understandingcited by 46
  19. 2025Can Large Vision Language Models Read Maps Like a Human?cited by 30
  20. 2025ReasonMap: Towards Fine-Grained Visual Reasoning from Transit Mapscited by 23
  21. 2025RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learningcited by 21
  22. 2026Visual Generation Unlocks Human-Like Reasoning through Multimodal World Modelscited by 19
  23. 2025Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMscited by 17
  24. 2025VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenescited by 4
  25. 2025MapTrace: Scalable Data Generation for Route Tracing on Mapscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SpatialCLI: Learning to Reason With Spatial Tools, Then Without Them
  2. 2026SpatiO: Adaptive Test-Time Orchestration of Vision-Language Agents for Spatial Reasoning
  3. 2026SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning
  4. 2026S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence
  5. 2026LAST: Leveraging Tools as Hints to Enhance Spatial Reasoning for Multimodal Large Language Models
  6. 2026From Reasoning Failures to Composable Video Spatial Intelligence
  7. 2026SpotAgent: Grounding Visual Geo-localization in Large Vision-Language Models through Agentic Reasoning
  8. 2026Active Exploring like a Pigeon: Reinforcing Spatial Reasoning via Agentic Vision-Language Models