mlsanityData to 7 Oct 2026

3D-Aware Vision-Language Models Fine-Tuning with Geometric Distillation

arXiv 2506.09883 ↗First posted 11 Jun 2025cs.CV cs.AIcited by 4

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-Language Models (VLMs) have shown remarkable performance on diverse visual and linguistic tasks, yet they remain fundamentally limited in their understanding of 3D spatial structures. We propose Geometric Distillation, a lightweight, annotation-free fine-tuning framework that injects human-inspired geometric cues into pretrained VLMs without modifying their architecture. By distilling (1) sparse correspondences, (2) relative depth relations, and (3) dense cost volumes from off-the-shelf 3D foundation models (e.g., MASt3R, VGGT), our method shapes representations to be geometry-aware while remaining compatible with natural image-text inputs. Through extensive evaluations on 3D vision-language reasoning and 3D perception benchmarks, our method consistently outperforms prior approaches, achieving improved 3D spatial reasoning with significantly lower computational cost. Our work demonstrates a scalable and efficient path to bridge 2D-trained VLMs with 3D understanding, opening up wider use in spatially grounded multimodal tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 60 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

39 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  2. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoning30 Oct
  3. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition20 Mar
  4. 2021PointCLIP: Point Cloud Understanding by CLIP4 Dec

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  7. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  8. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  9. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  10. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  11. 2024Grounding Image Matching in 3D with MASt3Rcited by 960
  12. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
23 more
  1. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 721
  2. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  3. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  4. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  5. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  6. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  7. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 258
  8. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  9. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  10. 2016Single-Image Depth Perception in the Wildcited by 179
  11. 2017Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistencycited by 177
  12. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  13. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
  14. 2023OnePose++: Keypoint-Free One-Shot Object Pose Estimation without CAD Modelscited by 114
  15. 2020Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrievalcited by 98
  16. 2018MonoGRNet: A Geometric Reasoning Network for Monocular 3D Object Localizationcited by 84
  17. 2024Improving 2D Feature Representations by 3D-Aware Fine-Tuningcited by 70
  18. 2024DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Videocited by 64
  19. 2022UniCLIP: Unified Framework for Contrastive Language-Image Pre-trainingcited by 54
  20. 2017Proposal Flow: Semantic Correspondences from Object Proposalscited by 46
  21. 2024Cross-View Completion Models are Zero-shot Correspondence Estimatorscited by 30
  22. 2024DepthCues: Evaluating Monocular Depth Perception in Large Vision Modelscited by 18
  23. 2024Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuningcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2026VGGT-Omegacited by 20
  2. 2025GLaD: Geometric Latent Distillation for Vision-Language-Action Modelscited by 10
  3. 2025Robust Driving QA through Metadata-Grounded Context and Task-Specific Promptscited by 1
  4. 2026What VGGT Knows About Overlap: Probing Geometric Foundation Models for Co-Visibilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025G2VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning
  2. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction
  3. 2026GeoWorld-VLM: Geometry from World Models for Vision-Language Models
  4. 2026Distilling 3D Spatial Reasoning into a Lightweight Vision-Language Model with CoT
  5. 2026Multi-View Relational Distillation for Spatial Reasoning with Vision-Language Models
  6. 2025Abstract 3D Perception for Spatial Intelligence in Vision-Language Models
  7. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D
  8. 20253D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding