mlsanityData to 7 Oct 2026

CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition

arXiv 2303.11313 ↗First posted 20 Mar 2023cs.CVcited by 77

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Vision-Language models like CLIP have been widely adopted for various tasks due to their impressive zero-shot capabilities. However, CLIP is not suitable for extracting 3D geometric features as it was trained on only images and text by natural language supervision. We work on addressing this limitation and propose a new framework termed CG3D (CLIP Goes 3D) where a 3D encoder is learned to exhibit zero-shot capabilities. CG3D is trained using triplets of pointclouds, corresponding rendered 2D images, and texts using natural language supervision. To align the features in a multimodal embedding space, we utilize contrastive loss on 3D features obtained from the 3D encoder, as well as visual and text features extracted from CLIP. We note that the natural images used to train CLIP and the rendered 2D images in CG3D have a distribution shift. Attempting to train the visual and text encoder to account for this shift results in catastrophic forgetting and a notable decrease in performance. To solve this, we employ prompt tuning and introduce trainable parameters in the input space to shift CLIP towards the 3D pre-training dataset utilized in CG3D. We extensively test our pre-trained CG3D framework and demonstrate its impressive capabilities in zero-shot, open scene understanding, and retrieval tasks. Further, it also serves as strong starting weights for fine-tuning in downstream 3D recognition tasks.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 80 references that are in the map. First posted on arXiv 20 Mar 2023.

Builds on

66 of 80 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021PointCLIP: Point Cloud Understanding by CLIP4 Dec

Other references · 65

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  10. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  6. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  7. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2022Visual Prompt Tuningcited by 1,674
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2020Point Transformercited by 1,462
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  13. 2016An overview of gradient descent optimization algorithmscited by 1,021
  14. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  15. 2018PointRCNN: 3D Object Proposal Generation and Detection from Point Cloudcited by 939
  16. 2019PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detectioncited by 890
  17. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  18. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  19. 2020PCT: Point cloud transformercited by 730
  20. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  21. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  22. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  23. 2018Deep Visual Domain Adaptation: A Surveycited by 634
  24. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  25. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 560
  26. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  27. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  28. 2020PointContrast: Unsupervised Pre-training for 3D Point Cloud Understandingcited by 502
  29. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  30. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  31. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  32. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  33. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  34. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  35. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 404
  36. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 376
  37. 2021PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Cloudscited by 211
  38. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 23; 2024: 20; 2025: 27; 2026: 7.

77 papers.

Sort:

By area:

  1. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  2. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
  3. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 225
  4. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  5. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  6. 2023Towards Open Vocabulary Learning: A Surveycited by 165
  7. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 130
  8. 2023Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidancecited by 123
  9. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  10. 2023SAI3D: Segment Any Instance in 3D Scenescited by 90
  11. 2023OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Datacited by 81
  12. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
38 more
  1. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  2. 2024The Double-Ellipsoid Geometry of CLIPcited by 42
  3. 2023SAM3D: Zero-Shot 3D Object Detection via Segment Anything Modelcited by 40
  4. 2023Connecting Multi-modal Contrastive Representationscited by 36
  5. 2024Dual Prototype Evolving for Test-Time Generalization of Vision-Language Modelscited by 36
  6. 2024Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Modelscited by 34
  7. 2023Bridging the Domain Gap: Self-Supervised 3D Scene Understanding with Foundation Modelscited by 26
  8. 2024TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understandingcited by 26
  9. 2023DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classificationcited by 23
  10. 2023Extending Multi-modal Contrastive Representationscited by 21
  11. 2023Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-trainingcited by 17
  12. 2023Back to 3D: Few-Shot 3D Keypoint Detection with Back-Projected 2D Featurescited by 17
  13. 2024ImOV3D: Learning Open-Vocabulary Point Clouds 3D Object Detection from Only 2D Imagescited by 17
  14. 2023Geometrically-driven Aggregation for Zero-shot 3D Point Cloud Understandingcited by 13
  15. 20243D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillationcited by 13
  16. 2023Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representationcited by 12
  17. 20243D Weakly Supervised Semantic Segmentation with 2D Vision-Language Guidancecited by 12
  18. 20253DAlign-DAER: Dynamic Attention Policy and Efficient Retrieval Strategy for Fine-grained 3D-Text Alignment at Scalecited by 12
  19. 2023MV-CLIP: Multi-View CLIP for Zero-shot 3D Shape Recognitioncited by 11
  20. 2024VG4D: Vision-Language Model Goes 4D Video Recognitioncited by 11
  21. 2025Ov3R: Open-Vocabulary Semantic 3D Reconstruction from RGB Videoscited by 10
  22. 2024OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Imagescited by 9
  23. 2025Label-Efficient LiDAR Semantic Segmentation with 2D-3D Vision Transformer Adapterscited by 8
  24. 2025PiSA: A Self-Augmented Data Engine and Training Strategy for 3D Understanding with Large Modelscited by 8
  25. 2025Details Matter for Indoor Open-vocabulary 3D Instance Segmentationcited by 8
  26. 2025Bidirectional Prototype-Reward co-Evolution for Test-Time Adaptation of Vision-Language Modelscited by 6
  27. 2023Fine-grained Text and Image Guided Point Cloud Completion with CLIP Modelcited by 5
  28. 2024CEIA: CLIP-Based Event-Image Alignment for Open-World Event-Based Understandingcited by 5
  29. 2024Multi-modal Relation Distillation for Unified 3D Representation Learningcited by 5
  30. 2025Describe, Adapt and Combine: Empowering CLIP Encoders for Open-set 3D Object Retrievalcited by 5
  31. 2023Open-Pose 3D Zero-Shot Learning: Benchmark and Challengescited by 4
  32. 2025LiDAR Remote Sensing Meets Weak Supervision: Concepts, Methods, and Perspectivescited by 4
  33. 2025A Review of 3D Object Detection with Vision-Language Modelscited by 4
  34. 20253D-Aware Vision-Language Models Fine-Tuning with Geometric Distillationcited by 4
  35. 2025Zero-Shot Skeleton-Based Action Recognition With Prototype-Guided Feature Alignmentcited by 4
  36. 2024Cross-Modal Self-Training: Aligning Images and Pointclouds to Learn Classification without Labelscited by 3
  37. 2024PEVA-Net: Prompt-Enhanced View Aggregation Network for Zero/Few-Shot Multi-View 3D Shape Recognitioncited by 3
  38. 2024Zero-Shot Dual-Path Integration Framework for Open-Vocabulary 3D Instance Segmentationcited by 3

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Parameter-Efficient CLIP Adaptation for 3D Understanding via Unified Tokenization
  2. 2023Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes
  3. 2023CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes
  4. 2026P3T: Prototypical Point-level Prompt Tuning with Enhanced Generalization for 3D Vision-Language Models
  5. 2025TeDA: Boosting Vision-Lanuage Models for Zero-Shot 3D Object Retrieval via Testing-time Distribution Alignment
  6. 2024GS-CLIP: Gaussian Splatting for Contrastive Language-Image-3D Pretraining from Real-World Data
  7. 2023Joint Representation Learning for Text and 3D Point Cloud
  8. 2024ZeroKey: Point-Level Reasoning and Zero-Shot 3D Keypoint Detection from Large Language Models