mlsanityData to 14 Oct 2026

3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding

arXiv 2506.01946 ↗First posted 2 Jun 2025cs.CVcited by 70

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Recent advances in scene understanding have leveraged multimodal large language models (MLLMs) for 3D reasoning by capitalizing on their strong 2D pretraining. However, the lack of explicit 3D data during MLLM pretraining limits 3D representation capability. In this paper, we investigate the 3D-awareness of MLLMs by evaluating multi-view correspondence and reveal a strong positive correlation between the quality of 3D-aware representation and downstream task performance. Motivated by this, we propose 3DRS, a framework that enhances MLLM 3D representation learning by introducing supervision from pretrained 3D foundation models. Our approach aligns MLLM visual features with rich 3D knowledge distilled from 3D models, effectively improving scene understanding. Extensive experiments across multiple benchmarks and MLLMs -- including visual grounding, captioning, and question answering -- demonstrate consistent performance gains. Project page: https://visual-ai.github.io/3drs

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 58 of its 69 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

58 of 69 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces18 Dec
  2. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoning18 Mar
  3. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
  4. 2022SQA3D: Situated Question Answering in 3D Scenes14 Oct

Other references · 54

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2024Qwen2 Technical Reportcited by 2,120
  12. 2025VGGT: Visual Geometry Grounded Transformercited by 1,539
38 more
  1. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  2. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 559
  3. 2024Long Context Transfer from Language to Visioncited by 515
  4. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  5. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  6. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  7. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  8. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  9. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  10. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  11. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  12. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awarenesscited by 183
  13. 2025FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Viewscited by 180
  14. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 177
  15. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  16. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  17. 2022Multi-View Transformer for 3D Visual Groundingcited by 140
  18. 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138
  19. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 138
  20. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
  21. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 122
  22. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priorscited by 120
  23. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  24. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 109
  25. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generationcited by 71
  26. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 71
  27. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  28. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 68
  29. 2022BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detectioncited by 66
  30. 2021Pri3D: Can 3D Priors Help 2D Representation Learning?cited by 61
  31. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  32. 2025Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuningcited by 51
  33. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 49
  34. 2024Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understandingcited by 49
  35. 20213D-to-2D Distillation for Indoor Scene Parsingcited by 48
  36. 2023DistillBEV: Boosting Multi-Camera 3D Object Detection with Cross-Modal Knowledge Distillationcited by 41
  37. 2024RadarDistill: Boosting Radar-based Object Detection Performance via Knowledge Distillation from LiDAR Featurescited by 28
  38. 2024Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuningcited by 14

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 16; 2026: 54.

70 papers.

Sort:

By area:

  1. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 134
  2. 2026A Pragmatic VLA Foundation Modelcited by 103
  3. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 96
  4. 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 65
  5. 2025FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Predictioncited by 38
  6. 2025Don't Blind Your VLA: Aligning Visual Representations for OOD Generalizationcited by 35
  7. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 29
  8. 2026VGGDrive: Empowering Vision-Language Models with Cross-View Geometric Grounding for Autonomous Drivingcited by 20
  9. 2025SpatialDreamer: Incentivizing Spatial Reasoning via Active Mental Imagerycited by 11
  10. 2025GLaD: Geometric Latent Distillation for Vision-Language-Action Modelscited by 11
  11. 2026ROCKET: Residual-Oriented Multi-Layer Alignment for Spatially-Aware Vision-Language-Action Modelscited by 11
  12. 2026Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Mapscited by 8
38 more
  1. 2026Generation Models Know Space: Unleashing Implicit 3D Priors for Scene Understandingcited by 7
  2. 2026Feed-Forward 3D Scene Modeling: A Problem-Driven Perspectivecited by 6
  3. 2025LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learningcited by 5
  4. 2025Vision-Language Memory for Spatial Reasoningcited by 5
  5. 2026AffordVLA: Injecting Affordance Representations into Vision-Language-Action Models via Implicit Feature Alignmentcited by 5
  6. 2026LARA: Latent Action Representation Alignment for Vision-Language-Action Modelscited by 5
  7. 2025OmniEVA: Embodied Versatile Planner via Task-Adaptive 3D-Grounded and Embodiment-aware Reasoningcited by 4
  8. 2025Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction-Reasoning Synergycited by 4
  9. 2026Unlocking Dense Metric Depth Estimation in VLMscited by 4
  10. 2025Vision-Centric Activation and Coordination for Multimodal Large Language Modelscited by 3
  11. 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modelingcited by 3
  12. 2026From Correspondence to Actions: Human-Like Multi-Image Spatial Reasoning in Multi-modal Large Language Modelscited by 3
  13. 2026EgoMind: Activating Spatial Cognition through Linguistic Reasoning in MLLMscited by 3
  14. 2026OpenWorldLib: A Unified Codebase and Definition of Advanced World Modelscited by 3
  15. 2026Dense Reward for Multi-View 3D Reasoning with Global Maps and Local Viewscited by 3
  16. 2025Reg3D: Reconstructive Geometry Instruction Tuning for 3D Scene Understandingcited by 2
  17. 20254D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillationcited by 2
  18. 2026MosaicThinker: On-Device Visual Spatial Reasoning for Embodied AI via Iterative Construction of Space Representationcited by 2
  19. 2026PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Modelscited by 2
  20. 2026Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Modelscited by 2
  21. 2026Proxy3D: Efficient 3D Representations for Vision-Language Models via Semantic Clustering and Alignmentcited by 2
  22. 2026VEGA: Visual Encoder Grounding Alignment for Spatially-Aware Vision-Language-Action Modelscited by 2
  23. 2026Reason, Then Re-reason: Cross-view Revisiting Improves Spatial Reasoningcited by 2
  24. 2026Modeling Cross-vision Synergy for Unified Large Vision Modelcited by 1
  25. 20263D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understandingcited by 1
  26. 2026Think, Act, Build: An Agentic Framework with Vision Language Models for Zero-Shot 3D Visual Groundingcited by 1
  27. 20263D-IDE: 3D Implicit Depth Emergentcited by 1
  28. 2026Learning Geometric Representations from Videos for Spatial Intelligent Multimodal Large Language Modelscited by 1
  29. 2026Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priorscited by 1
  30. 2026GeoAnchor: Collaborative Reasoning via Latent Decomposition for 3D Spatial Understandingcited by 1
  31. 2026ConsiSpace: Learning Geometric Consistency Matters for Video Spatial Reasoningcited by 1
  32. 2026ViSTR-Bench: Can MLLMs Reason from Continuous Visual Cues in Dynamic Scenes?cited by 1
  33. 2026SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Modelscited by 1
  34. 2026Beyond Single Object: Learning 3D Relations with Large Language Modelscited by 1
  35. 2026Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answeringcited by 1
  36. 2026SCP: Spatial Causal Prediction in Videocited by 0
  37. 2026Token Warping Helps MLLMs Look from Nearby Viewpointscited by 0
  38. 2026Dual-Pathway Geometry-Aware MLLM for Spatial Intelligencecited by 0

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025HMR3D: Hierarchical Multimodal Representation for 3D Scene Understanding with Large Vision-Language Model
  2. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding
  3. 2025MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation
  4. 2026PAR3D: A Unified 3D-MLLM with Part-Aware Representation for Scene Understanding
  5. 2026Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations
  6. 2026Enhancing MLLM Spatial Understanding via Active 3D Scene Exploration for Multi-Perspective Reasoning
  7. 20253D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding
  8. 2025Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs