# 3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding

arXiv: https://arxiv.org/abs/2507.23478 · HTML: https://mlsanity.com/paper/2507.23478/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 19 of its 71 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 19 of 19)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [3D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selection](https://mlsanity.com/paper/2204.06272/) (2022)
- [GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models](https://mlsanity.com/paper/2501.01428/) (2025)
- [D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Grounding](https://mlsanity.com/paper/2112.01551/) (2021)
- [LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences](https://mlsanity.com/paper/2412.01292/) (2024)
- [Contextual Modeling for 3D Dense Captioning on Point Clouds](https://mlsanity.com/paper/2210.03925/) (2022)
- [DC-Scene: Data-Centric Learning for 3D Scene Understanding](https://mlsanity.com/paper/2505.15232/) (2025)
- [Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotations](https://mlsanity.com/paper/2506.17545/) (2025)
- [3D CoCa: Contrastive Learners are 3D Captioners](https://mlsanity.com/paper/2504.09518/) (2025)
- [GaussianVLM: Scene-centric 3D Vision-Language Models using Language-aligned Gaussian Splats for Embodied Reasoning and Beyond](https://mlsanity.com/paper/2507.00886/) (2025)
- [Compile Scene Graphs with Reinforcement Learning](https://mlsanity.com/paper/2504.13617/) (2025)
- [DIP-R1: Deep Inspection and Perception with RL Looking Through and Understanding Complex Scenes](https://mlsanity.com/paper/2505.23179/) (2025)
- [Evaluating Gemini in an arena for learning](https://mlsanity.com/paper/2505.24477/) (2025)
- [Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models](https://mlsanity.com/paper/2507.16524/) (2025)
- [OpenScan: A Benchmark for Generalized Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2408.11030/) (2024)

## Built on by (most cited first, 30 of 47)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [VLA-R1: Enhancing Reasoning in Vision-Language-Action Models](https://mlsanity.com/paper/2510.01623/) (2025)
- [Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle](https://mlsanity.com/paper/2509.16679/) (2025)
- [MobileVLA-R1: Reinforcing Vision-Language-Action for Mobile Robots](https://mlsanity.com/paper/2511.17889/) (2025)
- [ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning](https://mlsanity.com/paper/2604.24300/) (2026)
- [Thinking in Dynamics: How Multimodal Large Language Models Perceive, Track, and Reason Dynamics in Physical 4D World](https://mlsanity.com/paper/2603.12746/) (2026)
- [EvoVLA: Self-Evolving Vision-Language-Action Model](https://mlsanity.com/paper/2511.16166/) (2025)
- [StereoAdapter: Adapting Stereo Depth Estimation to Underwater Scenes](https://mlsanity.com/paper/2509.16415/) (2025)
- [PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning](https://mlsanity.com/paper/2602.21992/) (2026)
- [VaseVQA: Multimodal Agent and Benchmark for Ancient Greek Pottery](https://mlsanity.com/paper/2509.17191/) (2025)
- [GeoWorld: Geometric World Models](https://mlsanity.com/paper/2602.23058/) (2026)
- [EgoLCD: Egocentric Video Generation with Long Context Diffusion](https://mlsanity.com/paper/2512.04515/) (2025)
- [AnyDepth: Depth Estimation Made Easy](https://mlsanity.com/paper/2601.02760/) (2026)
- [VaseVQA-3D: Benchmarking 3D VLMs on Ancient Greek Pottery](https://mlsanity.com/paper/2510.04479/) (2025)
- [Video2Layout: Recall and Reconstruct Metric-Grounded Cognitive Map for Spatial Reasoning](https://mlsanity.com/paper/2511.16160/) (2025)
- [3D-Layout-R1: Structured Reasoning for Language-Instructed Spatial Editing](https://mlsanity.com/paper/2603.22279/) (2026)
- [PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Grounding](https://mlsanity.com/paper/2512.20907/) (2025)
- [3D CoCa v2: Contrastive Learners with Test-Time Search for Generalizable Spatial Intelligence](https://mlsanity.com/paper/2601.06496/) (2026)
- [VIEW2SPACE: Studying Multi-View Visual Reasoning from Sparse Observations](https://mlsanity.com/paper/2603.16506/) (2026)
- [Grounded 3D-Aware Spatial Vision-Language Modeling](https://mlsanity.com/paper/2605.30307/) (2026)
- [DriveJudge: Rethinking Autonomous Driving Evaluation with Vision-Language Models](https://mlsanity.com/paper/2606.17362/) (2026)
- [Dense Reward for Multi-View 3D Reasoning with Global Maps and Local Views](https://mlsanity.com/paper/2606.23557/) (2026)
- [Multimodal Data Storage and Retrieval for Embodied AI: A Survey](https://mlsanity.com/paper/2508.13901/) (2025)
- [Smooth Operator: Smooth Verifiable Reward Activates Spatial Reasoning Ability of Vision-Language Model](https://mlsanity.com/paper/2601.07695/) (2026)
- [OnlineSI: Taming Large Language Model for Online 3D Understanding and Grounding](https://mlsanity.com/paper/2601.16538/) (2026)
- [UniMesh: Unifying 3D Mesh Understanding and Generation](https://mlsanity.com/paper/2604.17472/) (2026)
- [LMM-Track4D: Eliciting 4D Dynamic Reasoning in LMMs via Trajectory-Grounded Dialogue](https://mlsanity.com/paper/2605.19390/) (2026)
- [Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMs](https://mlsanity.com/paper/2606.01215/) (2026)
- [Error-Driven Scene Editing for 3D Grounding in Large Language Models](https://mlsanity.com/paper/2511.14086/) (2025)
- [3D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understanding](https://mlsanity.com/paper/2603.04976/) (2026)

## Abstract (arXiv)

Large vision-language models (VLMs) have made significant strides in 2D visual understanding tasks, sparking interest in extending these capabilities to 3D scene understanding. However, current 3D VLMs often struggle with robust reasoning and generalization due to limitations in high-quality spatial data and the static nature of viewpoint assumptions. To address these challenges, we propose 3D-R1, a foundation model that enhances the reasoning capabilities of 3D VLMs. Specifically, we first construct a high-quality synthetic dataset with CoT, named Scene-30K, leveraging existing 3D-VL datasets and a data engine based on Gemini 2.5 Pro. It serves as cold-start initialization data for 3D-R1. Moreover, we leverage RLHF policy such as GRPO in the reinforcement learning training process to enhance reasoning capabilities and introduce three reward functions: a perception reward, a semantic similarity reward and a format reward to maintain detection accuracy and answer semantic precision. Furthermore, we introduce a dynamic view selection strategy that adaptively chooses the most informative perspectives for 3D scene understanding. Extensive experiments demonstrate that 3D-R1 delivers an average improvement of 10% across various 3D scene benchmarks, highlighting its effectiveness in enhancing reasoning and generalization in 3D scene understanding. Code: https://github.com/AIGeeksGroup/3D-R1. Website: https://aigeeksgroup.github.io/3D-R1.