# N3D-VLM: Native 3D Grounding Enables Accurate Spatial Reasoning in Vision-Language Models

arXiv: https://arxiv.org/abs/2512.16561 · HTML: https://mlsanity.com/paper/2512.16561/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 33 of its 38 references that are in the map.
- First posted on arXiv 18 Dec 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 30 of 33)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs](https://mlsanity.com/paper/2406.16860/) (2024)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces](https://mlsanity.com/paper/2412.14171/) (2024)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details](https://mlsanity.com/paper/2507.02546/) (2025)
- [LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning](https://mlsanity.com/paper/2311.18651/) (2023)
- [LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness](https://mlsanity.com/paper/2409.18125/) (2024)
- [Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild](https://mlsanity.com/paper/2207.10660/) (2022)
- [Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding](https://mlsanity.com/paper/2412.00493/) (2024)
- [Sonata: Self-Supervised Learning of Reliable Point Representations](https://mlsanity.com/paper/2503.16429/) (2025)
- [GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models](https://mlsanity.com/paper/2501.01428/) (2025)
- [SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model](https://mlsanity.com/paper/2403.13064/) (2024)
- [SpatialLadder: Progressive Training for Spatial Reasoning in Vision-Language Models](https://mlsanity.com/paper/2510.08531/) (2025)
- [SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning](https://mlsanity.com/paper/2504.20024/) (2025)
- [SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding](https://mlsanity.com/paper/2412.04383/) (2024)
- [SpatialLM: Training Large Language Models for Structured Indoor Modeling](https://mlsanity.com/paper/2506.07491/) (2025)
- [Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness](https://mlsanity.com/paper/2504.01901/) (2025)
- [VLM-Grounder: A VLM Agent for Zero-Shot 3D Visual Grounding](https://mlsanity.com/paper/2410.13860/) (2024)
- [Detect Anything 3D in the Wild](https://mlsanity.com/paper/2504.07958/) (2025)

## Built on by (most cited first, 29 of 29)

- [VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration](https://mlsanity.com/paper/2601.22674/) (2026)
- [PoseVLA: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies](https://mlsanity.com/paper/2602.19710/) (2026)
- [Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3D](https://mlsanity.com/paper/2604.05212/) (2026)
- [LottieGPT: Tokenizing Vector Animation for Autoregressive Generation](https://mlsanity.com/paper/2604.11792/) (2026)
- [JAEGER: Joint 3D Audio-Visual Grounding and Reasoning in Simulated Physical Environments](https://mlsanity.com/paper/2602.18527/) (2026)
- [Unlocking Dense Metric Depth Estimation in VLMs](https://mlsanity.com/paper/2605.15876/) (2026)
- [HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models](https://mlsanity.com/paper/2603.25411/) (2026)
- [3D HAMSTER: Bridging Planning and Control in Hierarchical Vision Language Action Models through 3D Trajectory Guidance](https://mlsanity.com/paper/2606.31329/) (2026)
- [AlloSpatial: Agentic Harness Framework for Spatial Reasoning in Foundation Models](https://mlsanity.com/paper/2606.08952/) (2026)
- [3D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understanding](https://mlsanity.com/paper/2603.04976/) (2026)
- [EXPLORE-Bench: Egocentric Scene Prediction with Long-Horizon Reasoning](https://mlsanity.com/paper/2603.09731/) (2026)
- [World2Mind: Cognition Toolkit for Allocentric Spatial Reasoning in Foundation Models](https://mlsanity.com/paper/2603.09774/) (2026)
- [Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoning](https://mlsanity.com/paper/2604.06725/) (2026)
- [TAIHRI: Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction](https://mlsanity.com/paper/2604.08921/) (2026)
- [Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimation](https://mlsanity.com/paper/2605.25326/) (2026)
- [Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors](https://mlsanity.com/paper/2606.06891/) (2026)
- [ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoning](https://mlsanity.com/paper/2607.17599/) (2026)
- [3DHarnessBench: Probing Agentic 3D-to-Code Capabilities of Frontier Vision-Language Models](https://mlsanity.com/paper/2609.06535/) (2026)
- [UniFunc3D: Unified Active Spatial-Temporal Grounding for 3D Affordance Segmentation](https://mlsanity.com/paper/2603.23478/) (2026)
- [SpatialPoint: Spatial-aware Point Prediction for Embodied Localization](https://mlsanity.com/paper/2603.26690/) (2026)
- [Ground3D-LMM: Fine-Grained 3D Point Grounding and Spatial Reasoning with LMM](https://mlsanity.com/paper/2607.05493/) (2026)
- [Trace, Verify, and Correct: A Training-Free Framework for Spatial Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2608.04759/) (2026)
- [RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection](https://mlsanity.com/paper/2608.09147/) (2026)
- [Chain of Spatial Thoughts: Modality-Agnostic Spatial Grounding for Vision Language Models](https://mlsanity.com/paper/2608.10278/) (2026)
- [SpatialSpeak: QA-Native Reconstruction with Local and Global Context for Spatial Chain-of-Thought Reasoning](https://mlsanity.com/paper/2609.33616/) (2026)
- [SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Models](https://mlsanity.com/paper/2609.38716/) (2026)
- [Soft Spatial Reasoning](https://mlsanity.com/paper/2609.38717/) (2026)
- [FloorSAV: Elucidating Spatial Audio-Visual Context with 2D Floormap for AV-LLMs](https://mlsanity.com/paper/2610.11310/) (2026)
- [GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Models](https://mlsanity.com/paper/2610.18694/) (2026)

## Abstract (arXiv)

While current multimodal models can answer questions based on 2D images, they lack intrinsic 3D object perception, limiting their ability to comprehend spatial relationships and depth cues in 3D scenes. In this work, we propose N3D-VLM, a novel unified framework that seamlessly integrates native 3D object perception with 3D-aware visual reasoning, enabling both precise 3D grounding and interpretable spatial understanding. Unlike conventional end-to-end models that directly predict answers from RGB/RGB-D inputs, our approach equips the model with native 3D object perception capabilities, enabling it to directly localize objects in 3D space based on textual descriptions. Building upon accurate 3D object localization, the model further performs explicit reasoning in 3D, achieving more interpretable and structured spatial understanding. To support robust training for these capabilities, we develop a scalable data construction pipeline that leverages depth estimation to lift large-scale 2D annotations into 3D space, significantly increasing the diversity and coverage for 3D object grounding data, yielding over six times larger than the largest existing single-image 3D detection dataset. Moreover, the pipeline generates spatial question-answering datasets that target chain-of-thought (CoT) reasoning in 3D, facilitating joint training for both 3D object localization and 3D spatial reasoning. Experimental results demonstrate that our unified framework not only achieves state-of-the-art performance on 3D grounding tasks, but also consistently surpasses existing methods in 3D spatial reasoning in vision-language model.