# DepthLM: Metric Depth From Vision Language Models

arXiv: https://arxiv.org/abs/2509.25413 · HTML: https://mlsanity.com/paper/2509.25413/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)

## Builds on (most cited first, 29 of 29)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting](https://mlsanity.com/paper/2301.00493/) (2023)
- [SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities](https://mlsanity.com/paper/2401.12168/) (2024)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://mlsanity.com/paper/2410.02073/) (2024)
- [Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation](https://mlsanity.com/paper/2404.15506/) (2024)
- [UniDepth: Universal Monocular Metric Depth Estimation](https://mlsanity.com/paper/2403.18913/) (2024)
- [Seed1.5-VL Technical Report](https://mlsanity.com/paper/2505.07062/) (2025)
- [Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image](https://mlsanity.com/paper/2307.10984/) (2023)
- [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](https://mlsanity.com/paper/2406.01584/) (2024)
- [MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details](https://mlsanity.com/paper/2507.02546/) (2025)
- [Pixtral 12B](https://mlsanity.com/paper/2410.07073/) (2024)
- [UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler](https://mlsanity.com/paper/2502.20110/) (2025)
- [SpatialBot: Precise Spatial Understanding with Vision Language Models](https://mlsanity.com/paper/2406.13642/) (2024)
- [Evaluation of CNN-based Single-Image Depth Estimation Methods](https://mlsanity.com/paper/1805.01328/) (2018)
- [Waymo Open Dataset: Panoramic Video Panoptic Segmentation](https://mlsanity.com/paper/2206.07704/) (2022)
- [Multi-SpatialMLLM: Multi-Frame Spatial Understanding with Multi-Modal Large Language Models](https://mlsanity.com/paper/2505.17015/) (2025)
- [Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling](https://mlsanity.com/paper/2405.14578/) (2024)

## Built on by (most cited first, 30 of 30)

- [Image Generators are Generalist Vision Learners](https://mlsanity.com/paper/2604.20329/) (2026)
- [VGGT-Omega](https://mlsanity.com/paper/2605.15195/) (2026)
- [SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL](https://mlsanity.com/paper/2512.04069/) (2025)
- [Towards Spatial Trace with Reasoning in Vision-Language Models for Robotics](https://mlsanity.com/paper/2512.13660/) (2025)
- [Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models](https://mlsanity.com/paper/2606.11324/) (2026)
- [UniSER: A Foundation Model for Unified Soft Effects Removal](https://mlsanity.com/paper/2511.14183/) (2025)
- [Unlocking Dense Metric Depth Estimation in VLMs](https://mlsanity.com/paper/2605.15876/) (2026)
- [DepthFocus: Controllable Depth Estimation for See-Through Scenes](https://mlsanity.com/paper/2511.16993/) (2025)
- [GAP-MLLM: Geometry-Aligned Pre-training for Activating 3D Spatial Perception in Multimodal Large Language Models](https://mlsanity.com/paper/2603.16461/) (2026)
- [Perceptio: Perception Enhanced Vision Language Models via Spatial Token Generation](https://mlsanity.com/paper/2603.18795/) (2026)
- [Grounded 3D-Aware Spatial Vision-Language Modeling](https://mlsanity.com/paper/2605.30307/) (2026)
- [Stepping VLMs onto the Court: Benchmarking Spatial Intelligence in Sports](https://mlsanity.com/paper/2603.09896/) (2026)
- [WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigation](https://mlsanity.com/paper/2603.10703/) (2026)
- [Linking Perception, Confidence and Accuracy in MLLMs](https://mlsanity.com/paper/2603.12149/) (2026)
- [VLM3: Vision Language Models Are Native 3D Learners](https://mlsanity.com/paper/2605.30561/) (2026)
- [Ill-Posed by Design: Probing Evidence Use in VLMs](https://mlsanity.com/paper/2606.24335/) (2026)
- [TAIHRI: Task-Aware 3D Human Keypoints Localization for Close-Range Human-Robot Interaction](https://mlsanity.com/paper/2604.08921/) (2026)
- [Consistent Yet Wrong: Evidence Insensitivity in Spatial Vision-Language Models](https://mlsanity.com/paper/2606.02742/) (2026)
- [SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models](https://mlsanity.com/paper/2608.01899/) (2026)
- [DenseMLLM: Standard Multimodal LLMs for Dense Prediction](https://mlsanity.com/paper/2602.14134/) (2026)
- [M2-VLA: Boosting Vision-Language Models for Generalizable Manipulation via Layer Mixture and Meta-Skills](https://mlsanity.com/paper/2604.24182/) (2026)
- [SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models](https://mlsanity.com/paper/2605.31597/) (2026)
- [STAR-VLM: Spatiotemporal Grounding Vision-Language Models for Motion and Velocity Estimation via Automotive Radar Supervision](https://mlsanity.com/paper/2608.01535/) (2026)
- [ZIL: Zero-shot Image-to-LiDAR Registration](https://mlsanity.com/paper/2609.22716/) (2026)
- [Metric-Bench: Exploring In-context Spatial Metric Reasoning in VLMs for Indoor Scenes](https://mlsanity.com/paper/2609.25841/) (2026)
- [DepthEvidence: Unifying Metric Depth Prediction and Geometric Reasoning in Multimodal Language Models](https://mlsanity.com/paper/2609.31103/) (2026)
- [SpatialSpeak: QA-Native Reconstruction with Local and Global Context for Spatial Chain-of-Thought Reasoning](https://mlsanity.com/paper/2609.33616/) (2026)
- [SpatialCORE: Confidence-Aware Grounded Spatial Reasoning in Large Vision--Language Models](https://mlsanity.com/paper/2609.38716/) (2026)
- [Soft Spatial Reasoning](https://mlsanity.com/paper/2609.38717/) (2026)
- [Render to Reason: Novel-View Semantic Prediction Improves Spatial Understanding in VLMs](https://mlsanity.com/paper/2610.05417/) (2026)

## Abstract (arXiv)

Vision language models (VLMs) can flexibly address various vision tasks through text interactions. Although successful in semantic understanding, state-of-the-art VLMs including GPT-5 still struggle in understanding 3D from 2D inputs. On the other hand, expert pure vision models achieve super-human accuracy in metric depth estimation, a key 3D understanding task. However, they require task-specific architectures and losses. Such difference motivates us to ask: Can VLMs reach expert-level accuracy without architecture or loss change? We take per-pixel metric depth estimation as the representative task and show that the answer is yes! Surprisingly, comprehensive analysis shows that text-based supervised-finetuning with sparse labels is sufficient for VLMs to unlock strong 3D understanding, no dense prediction head or complex regression/regularization loss is needed. The bottleneck for VLMs lies actually in pixel reference and cross-dataset camera ambiguity, which we address through visual prompting and intrinsic-conditioned augmentation. With much smaller models, our method DepthLM surpasses the accuracy of most advanced VLMs by over 2x, making VLMs for the first time comparable with pure vision models. Interestingly, without explicit enforcement during training, VLMs trained with DepthLM naturally avoids over-smoothing, having much fewer flying points at boundary regions than pure vision models. The simplicity of DepthLM also enables a single VLM to cover various 3D tasks beyond metric depth. Our code and model will be released at the link below.