# NuScenes-SpatialQA: A Spatial Understanding and Reasoning Benchmark for Vision-Language Models in Autonomous Driving

arXiv: https://arxiv.org/abs/2504.03164 · HTML: https://mlsanity.com/paper/2504.03164/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Chain-of-thought reasoning for driving, one of 17 directions in LLM reasoning.
- Builds on 11 of its 57 references that are in the map.
- First posted on arXiv 4 Apr 2025.

## Placed in

- LLM reasoning / [Chain-of-thought reasoning for driving](https://mlsanity.com/lr/chain-of-thought-reasoning-for-driving/)
- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)
- End-to-end autonomous driving / [VLMs for autonomous driving](https://mlsanity.com/driving/vlms-for-autonomous-driving/)

## Builds on (most cited first, 11 of 11)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [EMMA: End-to-End Multimodal Model for Autonomous Driving](https://mlsanity.com/paper/2410.23262/) (2024)
- [LingoQA: Visual Question Answering for Autonomous Driving](https://mlsanity.com/paper/2312.14115/) (2023)
- [RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Model](https://mlsanity.com/paper/2402.10828/) (2024)
- [CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving](https://mlsanity.com/paper/2408.10845/) (2024)
- [Multi-Frame, Lightweight & Efficient Vision-Language Models for Question Answering in Autonomous Driving](https://mlsanity.com/paper/2403.19838/) (2024)
- [STAMP: Scalable Task And Model-agnostic Collaborative Perception](https://mlsanity.com/paper/2501.18616/) (2025)
- [AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving](https://mlsanity.com/paper/2412.15206/) (2024)
- [OmniDrive: A Holistic Vision-Language Dataset for Autonomous Driving with Counterfactual Reasoning](https://mlsanity.com/paper/2405.01533/) (2024)
- [Physically Analyzable AI-Based Nonlinear Platoon Dynamics Modeling During Traffic Oscillation: A Koopman Approach](https://mlsanity.com/paper/2406.14696/) (2024)

## Built on by (most cited first, 30 of 36)

- [AutoVLA: A Vision-Language-Action Model for End-to-End Autonomous Driving with Adaptive Reasoning and Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.13757/) (2025)
- [Alpamayo-R1: Bridging Reasoning and Action Prediction for Generalizable Autonomous Driving in the Long Tail](https://mlsanity.com/paper/2511.00088/) (2025)
- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2510.27606/) (2025)
- [STSBench: A Spatio-temporal Scenario Benchmark for Multi-modal Large Language Models in Autonomous Driving](https://mlsanity.com/paper/2506.06218/) (2025)
- [RefAV: Towards Planning-Centric Scenario Mining](https://mlsanity.com/paper/2505.20981/) (2025)
- [From Pixels to Places: A Systematic Benchmark for Evaluating Image Geolocalization Ability in Large Language Models](https://mlsanity.com/paper/2508.01608/) (2025)
- [Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models](https://mlsanity.com/paper/2604.01259/) (2026)
- [Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving](https://mlsanity.com/paper/2511.19221/) (2025)
- [Unifying Language-Action Understanding and Generation for Autonomous Driving](https://mlsanity.com/paper/2603.01441/) (2026)
- [Generative AI for Autonomous Driving: Frontiers and Opportunities](https://mlsanity.com/paper/2505.08854/) (2025)
- [RSGround-R1: Rethinking Remote Sensing Visual Grounding through Spatial Reasoning](https://mlsanity.com/paper/2601.21634/) (2026)
- [Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluation](https://mlsanity.com/paper/2602.11635/) (2026)
- [SpatialEvo: Self-Evolving Spatial Intelligence via Deterministic Geometric Environments](https://mlsanity.com/paper/2604.14144/) (2026)
- [Automated Vehicles Should be Connected with Natural Language](https://mlsanity.com/paper/2507.01059/) (2025)
- [MonoSR: Open-Vocabulary Spatial Reasoning from Monocular Images](https://mlsanity.com/paper/2511.19119/) (2025)
- [More than the Sum: Panorama-Language Models for Adverse Omni-Scenes](https://mlsanity.com/paper/2603.09573/) (2026)
- [Embodied4C: Measuring What Matters for Embodied Vision-Language Navigation](https://mlsanity.com/paper/2512.18028/) (2025)
- [ADV-0: Closed-Loop Min-Max Adversarial Training for Long-Tail Robustness in Autonomous Driving](https://mlsanity.com/paper/2603.15221/) (2026)
- [Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study](https://mlsanity.com/paper/2510.21160/) (2025)
- [nuScenes Revisited: Progress and Challenges in Autonomous Driving](https://mlsanity.com/paper/2512.02448/) (2025)
- [Listen, Look, Drive: Coupling Audio Instructions for User-aware VLA-based Autonomous Driving](https://mlsanity.com/paper/2601.12142/) (2026)
- [DRIVESPATIAL: A Benchmark for Spatiotemporal Intelligence in VLMs for Autonomous Driving](https://mlsanity.com/paper/2605.23176/) (2026)
- [Understanding Space Is Rocket Science -- Only Top Reasoning Models Can Solve Spatial Understanding Tasks](https://mlsanity.com/paper/2509.02175/) (2025)
- [ObsDriveBench: Benchmarking Multimodal Understanding under Adverse Weather with Observability Awareness](https://mlsanity.com/paper/2607.23537/) (2026)
- [Deferred Exposure of Future Trajectories for Verifiable Reasoning in Autonomous Driving VLMs](https://mlsanity.com/paper/2608.01755/) (2026)
- [From Steering to Pedalling: Do Autonomous Driving VLMs Generalize to Cyclist-Assistive Spatial Perception and Planning?](https://mlsanity.com/paper/2602.10771/) (2026)
- [Information Coordination as a Bridge: A Neuro-Symbolic Architecture for Reliable Autonomous Driving Scene Understanding](https://mlsanity.com/paper/2605.04475/) (2026)
- [SUMO: Segment and Track Any Motion with Nonlinear State Space Models](https://mlsanity.com/paper/2606.29861/) (2026)
- [FactorDrive: Adaptive Multi-Step Reasoning Driven by Planning-Critical Factors for End-to-End Autonomous Driving](https://mlsanity.com/paper/2608.09591/) (2026)

## Abstract (arXiv)

Recent advancements in Vision-Language Models (VLMs) have demonstrated strong potential for autonomous driving tasks. However, their spatial understanding and reasoning-key capabilities for autonomous driving-still exhibit significant limitations. Notably, none of the existing benchmarks systematically evaluate VLMs' spatial reasoning capabilities in driving scenarios. To fill this gap, we propose NuScenes-SpatialQA, the first large-scale ground-truth-based Question-Answer (QA) benchmark specifically designed to evaluate the spatial understanding and reasoning capabilities of VLMs in autonomous driving. Built upon the NuScenes dataset, the benchmark is constructed through an automated 3D scene graph generation pipeline and a QA generation pipeline. The benchmark systematically evaluates VLMs' performance in both spatial understanding and reasoning across multiple dimensions. Using this benchmark, we conduct extensive experiments on diverse VLMs, including both general and spatial-enhanced models, providing the first comprehensive evaluation of their spatial capabilities in autonomous driving. Surprisingly, the experimental results show that the spatial-enhanced VLM outperforms in qualitative QA but does not demonstrate competitiveness in quantitative QA. In general, VLMs still face considerable challenges in spatial understanding and reasoning.