mlsanityData to 7 Oct 2026

DepthLM: Metric Depth From Vision Language Models

arXiv 2509.25413 ↗First posted 29 Sep 2025cs.CVcited by 29

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision language models (VLMs) can flexibly address various vision tasks through text interactions. Although successful in semantic understanding, state-of-the-art VLMs including GPT-5 still struggle in understanding 3D from 2D inputs. On the other hand, expert pure vision models achieve super-human accuracy in metric depth estimation, a key 3D understanding task. However, they require task-specific architectures and losses. Such difference motivates us to ask: Can VLMs reach expert-level accuracy without architecture or loss change? We take per-pixel metric depth estimation as the representative task and show that the answer is yes! Surprisingly, comprehensive analysis shows that text-based supervised-finetuning with sparse labels is sufficient for VLMs to unlock strong 3D understanding, no dense prediction head or complex regression/regularization loss is needed. The bottleneck for VLMs lies actually in pixel reference and cross-dataset camera ambiguity, which we address through visual prompting and intrinsic-conditioned augmentation. With much smaller models, our method DepthLM surpasses the accuracy of most advanced VLMs by over 2x, making VLMs for the first time comparable with pure vision models. Interestingly, without explicit enforcement during training, VLMs trained with DepthLM naturally avoids over-smoothing, having much fewer flying points at boundary regions than pure vision models. The simplicity of DepthLM also enables a single VLM to cover various 3D tasks beyond metric depth. Our code and model will be released at the link below.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 36 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

29 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models22 May
  2. 2024SpatialBot: Precise Spatial Understanding with Vision Language Models19 Jun
  3. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models3 Jun
  4. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan

Other references · 25

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  7. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  9. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  10. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 701
  11. 2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
  12. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
13 more
  1. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  2. 20193D Packing for Self-Supervised Monocular Depth Estimationcited by 437
  3. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 395
  4. 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 389
  5. 2024UniDepth: Universal Monocular Metric Depth Estimationcited by 375
  6. 2025Seed1.5-VL Technical Reportcited by 367
  7. 2023Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Imagecited by 357
  8. 2025MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Detailscited by 250
  9. 2024Pixtral 12Bcited by 174
  10. 2025UniDepthV2: Universal Monocular Metric Depth Estimation Made Simplercited by 170
  11. 2018Evaluation of CNN-based Single-Image Depth Estimation Methodscited by 137
  12. 2022Waymo Open Dataset: Panoramic Video Panoptic Segmentationcited by 57
  13. 2024Surge Phenomenon in Optimal Learning Rate and Batch Size Scalingcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 25.

29 papers.

Sort:

By area:

  1. 2026Image Generators are Generalist Vision Learnerscited by 40
  2. 2025SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RLcited by 20
  3. 2026VGGT-Omegacited by 20
  4. 2025Towards Spatial Trace with Reasoning in Vision-Language Models for Roboticscited by 12
  5. 2025UniSER: A Foundation Model for Unified Soft Effects Removalcited by 5
  6. 2026Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Modelscited by 5
  7. 2025DepthFocus: Controllable Depth Estimation for See-Through Scenescited by 3
  8. 2026Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sportscited by 2
  9. 2026WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigationcited by 2
  10. 2026Linking Perception, Confidence and Accuracy in MLLMscited by 2
  11. 2026Perceptio: Perception Enhanced Vision Language Models via Spatial Token Generationcited by 2
  12. 2026Unlocking Dense Metric Depth Estimation in VLMscited by 2
17 more
  1. 2026VLM3: Vision Language Models Are Native 3D Learnerscited by 2
  2. 2026Ill-Posed by Design: Probing Evidence Use in VLMscited by 2
  3. 2026GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Modelscited by 1
  4. 2026TAIHRI: Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interactioncited by 1
  5. 2026Grounded 3D-Aware Spatial Vision-Language Modelingcited by 1
  6. 2026Consistent Yet Wrong: Evidence Insensitivity in Spatial Vision-Language Modelscited by 1
  7. 2026DenseMLLM: Standard Multimodal LLMs for Dense Predictioncited by 0
  8. 2026M2-VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skillscited by 0
  9. 2026SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Modelscited by 0
  10. 2026STAR-VLM: Spatiotemporal Grounding Vision-Language Models for Motion and Velocity Estimation via Automotive Radar Supervisioncited by 0
  11. 2026SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Modelscited by 0
  12. 2026ZIL: Zero-shot Image-to-LiDAR Registrationcited by 0
  13. 2026Metric-Bench: Exploring In-context Spatial Metric Reasoning in VLMs for Indoor Scenescited by 0
  14. 2026DepthEvidence: Unifying Metric Depth Prediction and Geometric Reasoning in Multimodal Language Modelscited by 0
  15. 2026SpatialSpeak: QA-Native Reconstruction with Local and Global Context for Spatial Chain-of-Thought Reasoningcited by 0
  16. 2026SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Modelscited by 0
  17. 2026Soft Spatial Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Hidden in plain sight: VLMs overlook their visual representations
  2. 2025SD-VLM: Spatial Measuring and Understanding with Depth-Encoded Vision-Language Models
  3. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction
  4. 2026An overview of 3D Vision-Language Models
  5. 2026Seeing Is Not Measuring: Tool-Augmented Metric Spatial Reasoning for Vision-Language Models
  6. 2023Uni3DL: Unified Model for 3D and Language Understanding
  7. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs
  8. 2024An Introduction to Vision-Language Modeling