mlsanityData to 14 Oct 2026

An overview of 3D Vision-Language Models

arXiv 2609.05583 ↗First posted 4 Sep 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-Language Models (VLMs) are reshaping computer vision by aligning visual and textual embeddings, allowing models to recognize visual concepts and reason about them using natural language. Traditional 3D deep-learning models, however, are typically trained for specific tasks, such as classification, segmentation, or detection, and do not naturally support cross-modal retrieval from their embedding spaces using text or images as queries. To address this issue, Contrastive Language-Image Pretraining (CLIP)-based methods align 3D embeddings with pretrained image and text representations, giving rise to 3D Vision-Language Models (3D VLMs) that support zero-shot classification, cross-modal retrieval, and open-vocabulary recognition of 3D shapes. This tutorial provides an overview of 3D VLMs, ranging from basic definitions of 3D representations and their encoding into embeddings to cross-modal contrastive alignment, modern multimodal frameworks, and 3D Vision-Large Language Models (3D VLLMs). We present the main definitions of contrastive learning for multimodal embedding alignment and highlight recent advances in language-guided 3D Gaussian splatting, 3D shape generation, and embodied AI for robotics.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 72 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

65 of 72 references.

Closest ancestors · 9

Same direction, newest first

  1. 20263D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning29 Aug
  2. 2026PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Models28 Feb
  3. 2025SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretraining23 Mar
  4. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models16 May
  5. 2024OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images25 Apr
  6. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interaction27 Feb
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  8. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition20 Mar
  9. 2021PointCLIP: Point Cloud Understanding by CLIP4 Dec

Other references · 56

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  4. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  5. 2025Qwen3-VL Technical Reportcited by 2,793
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  7. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  8. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  9. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  10. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  11. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  12. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
38 more
  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  2. 2022A Generalist Agentcited by 808
  3. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  4. 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 770
  5. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 653
  6. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  7. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 565
  8. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 459
  9. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 393
  10. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  11. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
  12. 2023MeshGPT: Generating Triangle Meshes with Decoder-Only Transformerscited by 259
  13. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 237
  14. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 228
  15. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 208
  16. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 202
  17. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187
  18. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 184
  19. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 160
  20. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 145
  21. 2018MeshNet: Mesh Neural Network for 3D Shape Representationcited by 95
  22. 2025Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representationscited by 45
  23. 2025Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splattingcited by 31
  24. 2024TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understandingcited by 27
  25. 2022Exploring Differential Geometry in Neural Implicitscited by 26
  26. 2024GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splattingcited by 25
  27. 2026Utonia: Toward One Encoder for All Point Cloudscited by 21
  28. 2022Neural Implicit Surface Evolutioncited by 18
  29. 2022EPCL: Frozen CLIP Transformer is An Efficient Point Cloud Encodercited by 18
  30. 2023Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-trainingcited by 18
  31. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 15
  32. 2024Duoduo CLIP: Efficient 3D Understanding with Multi-View Imagescited by 12
  33. 2025Foundational Models for 3D Point Clouds: A Survey and Outlookcited by 12
  34. 2025Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answeringcited by 11
  35. 2024CLIP-GS: Unifying Vision-Language Representation with 3D Gaussian Splattingcited by 7
  36. 2026MeshFlow: Efficient Artistic Mesh Generation via MeshVAE and Flow-based Diffusion Transformercited by 7
  37. 2025UniGS: Unified Language-Image-3D Pretraining with Gaussian Splattingcited by 5
  38. 2025A Review of 3D Object Detection with Vision-Language Modelscited by 4

50 most cited of 56 All 56 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026VLM3: Vision Language Models Are Native 3D Learners
  2. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data
  3. 2025Image Recognition with Vision and Language Embeddings of VLMs
  4. 20243D Scene Graph Guided Vision-Language Pre-training
  5. 2023Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding
  6. 2025DepthLM: Metric Depth From Vision Language Models
  7. 2025Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors
  8. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone