Perception-Aware Multimodal Spatial Reasoning from Monocular Images
On the map: LLM reasoning → Spatial and embodied reasoning
Abstract (arXiv)
Spatial reasoning from monocular images is essential for autonomous driving, yet current Vision-Language Models (VLMs) still struggle with fine-grained geometric perception, particularly under large scale variation and ambiguous object appearance. We propose a simple yet effective perception-aware multimodal reasoning framework that equips VLMs with explicit object-centric grounding ability. Instead of relying on textual bounding-box outputs, each referred object is represented using all Visual Reference Tokens (VRTs) within its spatial extent, enabling visual evidence and textual reasoning to be processed jointly in a unified token space. To further strengthen cross-modal interaction, we construct a Multimodal Chain-of-Thought (MM-CoT) dataset that injects aligned visual and textual reasoning signals. A deterministic ordering strategy is introduced to make supervision over inherently unordered VRT sets fully compatible with the VLM's autoregressive next-token prediction. With only standard supervised fine-tuning, our method achieves substantial improvements on the SURDS benchmark, outperforming previous approaches - including those using RL-based post-training - by a large margin across both single-object and multi-object tasks. These results demonstrate that accurate perception and multimodal reasoning are mutually reinforcing, and together form the key to robust spatial understanding in challenging monocular driving scenarios.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 71 references that are in the map. First posted on arXiv 7 Mar 2026.
Builds on
25 of 71 references.
Closest ancestors · 1
Same direction, newest first
Other references · 24
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2023GPT-4 Technical Reportcited by 18,132
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2023Qwen Technical Reportcited by 3,273
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
- 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
- 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
- 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
- 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
- 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
12 more
- 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
- 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
- 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
- 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 501
- 2023Emu: Generative Pretraining in Multimodalitycited by 156
- 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
- 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
- 2025Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learningcited by 43
- 2024MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Taskscited by 41
- 2025VisRL: Intention-Driven Visual Perception via Reinforced Reasoningcited by 39
- 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
- 2025Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMscited by 20
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
- 2025VGR: Visual Grounded Reasoning
- 2025Spatial-ViLT: Enhancing Visual Spatial Reasoning through Multi-Task Learning
- 2026Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models
- 2026Chain of Spatial Thoughts: Modality-Agnostic Spatial Grounding for Vision Language Models
- 2026Faithful GRPO: Improving Visual Spatial Reasoning in Multimodal Language Models via Constrained Policy Optimization
- 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens
- 2026When More Is Less: A Systematic Analysis of Spatial and Commonsense Information for Visual Spatial Reasoning