# Visual Agentic AI for Spatial Reasoning with a Dynamic API

arXiv: https://arxiv.org/abs/2502.06787 · HTML: https://mlsanity.com/paper/2502.06787/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 9 of its 45 references that are in the map.
- First posted on arXiv 10 Feb 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 9 of 9)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://mlsanity.com/paper/2409.17146/) (2024)
- [Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts](https://mlsanity.com/paper/2111.08276/) (2021)
- [What's "up" with vision-language models? Investigating their struggle with spatial reasoning](https://mlsanity.com/paper/2310.19785/) (2023)
- [LAB-Bench: Measuring Capabilities of Language Models for Biology Research](https://mlsanity.com/paper/2407.10362/) (2024)

## Built on by (most cited first, 30 of 43)

- [Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing](https://mlsanity.com/paper/2506.09965/) (2025)
- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [3D Aware Region Prompted Vision Language Model](https://mlsanity.com/paper/2509.13317/) (2025)
- [SITE: towards Spatial Intelligence Thorough Evaluation](https://mlsanity.com/paper/2505.05456/) (2025)
- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [Think3D: Thinking with Space for Spatial Reasoning](https://mlsanity.com/paper/2601.13029/) (2026)
- [Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods](https://mlsanity.com/paper/2511.15722/) (2025)
- [Is CLIP ideal? No. Can we fix it? Yes!](https://mlsanity.com/paper/2503.08723/) (2025)
- [ST-Think: How Multimodal Large Language Models Reason About 4D Worlds from Ego-Centric Videos](https://mlsanity.com/paper/2503.12542/) (2025)
- [SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL](https://mlsanity.com/paper/2512.04069/) (2025)
- [SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning](https://mlsanity.com/paper/2606.13673/) (2026)
- [pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoning](https://mlsanity.com/paper/2603.00905/) (2026)
- [RefAV: Towards Planning-Centric Scenario Mining](https://mlsanity.com/paper/2505.20981/) (2025)
- [Euclid's Gift: Enhancing Spatial Perception and Reasoning in Vision-Language Models via Geometric Surrogate Tasks](https://mlsanity.com/paper/2509.24473/) (2025)
- [S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence](https://mlsanity.com/paper/2606.20515/) (2026)
- [Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMs](https://mlsanity.com/paper/2506.04220/) (2025)
- [Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Driving](https://mlsanity.com/paper/2609.00111/) (2026)
- [Agentic Retoucher for Text-To-Image Generation](https://mlsanity.com/paper/2601.02046/) (2026)
- [ESCA: Contextualizing Embodied Agents via Scene-Graph Generation](https://mlsanity.com/paper/2510.15963/) (2025)
- [SAGE: Training Smart Any-Horizon Agents for Long Video Reasoning with Reinforcement Learning](https://mlsanity.com/paper/2512.13874/) (2025)
- [Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluation](https://mlsanity.com/paper/2602.11635/) (2026)
- [SUPERGLASSES: Benchmarking Vision Language Models as Intelligent Agents for AI Smart Glasses](https://mlsanity.com/paper/2602.22683/) (2026)
- [Same or Not? Enhancing Visual Perception in Vision-Language Models](https://mlsanity.com/paper/2512.23592/) (2025)
- [Aligning Text, Images, and 3D Structure Token-by-Token](https://mlsanity.com/paper/2506.08002/) (2025)
- [No Labels, No Problem: Training Visual Reasoners with Multimodal Verifiers](https://mlsanity.com/paper/2512.08889/) (2025)
- [Grounded 3D-Aware Spatial Vision-Language Modeling](https://mlsanity.com/paper/2605.30307/) (2026)
- [DocReward: A Document Reward Model for Structuring and Stylizing](https://mlsanity.com/paper/2510.11391/) (2025)
- [CompAgent: An Agentic Framework for Visual Compliance Verification](https://mlsanity.com/paper/2511.00171/) (2025)
- [Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction](https://mlsanity.com/paper/2607.08233/) (2026)
- [SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models](https://mlsanity.com/paper/2608.01899/) (2026)

## Abstract (arXiv)

Visual reasoning -- the ability to interpret the visual world -- is crucial for embodied agents that operate within three-dimensional scenes. Progress in AI has led to vision and language models capable of answering questions from images. However, their performance declines when tasked with 3D spatial reasoning. To tackle the complexity of such reasoning problems, we introduce an agentic program synthesis approach where LLM agents collaboratively generate a Pythonic API with new functions to solve common subproblems. Our method overcomes limitations of prior approaches that rely on a static, human-defined API, allowing it to handle a wider range of queries. To assess AI capabilities for 3D understanding, we introduce a new benchmark of queries involving multiple steps of grounding and inference. We show that our method outperforms prior zero-shot models for visual reasoning in 3D and empirically validate the effectiveness of our agentic framework for 3D spatial reasoning tasks. Project website: https://glab-caltech.github.io/vadar/