mlsanityData to 14 Oct 2026

DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classification

arXiv 2305.15957 ↗First posted 25 May 2023cs.CVcited by 23

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Large pre-trained models have had a significant impact on computer vision by enabling multi-modal learning, where the CLIP model has achieved impressive results in image classification, object detection, and semantic segmentation. However, the model's performance on 3D point cloud processing tasks is limited due to the domain gap between depth maps from 3D projection and training images of CLIP. This paper proposes DiffCLIP, a new pre-training framework that incorporates stable diffusion with ControlNet to minimize the domain gap in the visual branch. Additionally, a style-prompt generation module is introduced for few-shot tasks in the textual branch. Extensive experiments on the ModelNet10, ModelNet40, and ScanObjectNN datasets show that DiffCLIP has strong abilities for 3D understanding. By using stable diffusion and style-prompt generation, DiffCLIP achieves an accuracy of 43.2% for zero-shot classification on OBJBG of ScanObjectNN, which is state-of-the-art performance, and an accuracy of 80.6% for zero-shot classification on ModelNet10, which is comparable to state-of-the-art performance.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 43 of its 57 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

43 of 57 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
31 more
  1. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  2. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  3. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  4. 2020Point Transformercited by 1,470
  5. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  6. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  7. 2021Robust fine-tuning of zero-shot modelscited by 797
  8. 2020PCT: Point cloud transformercited by 732
  9. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 565
  10. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  11. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 463
  12. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  13. 2022Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Modelscited by 382
  14. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 342
  15. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 187
  16. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
  17. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 160
  18. 2023CLIPood: Generalizing CLIP to Out-of-Distributionscited by 96
  19. 2022FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objectscited by 91
  20. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  21. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69
  22. 2020End-to-End Learning Local Multi-view Descriptors for 3D Point Cloudscited by 47
  23. 2022TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulationcited by 47
  24. 2019Personalization of End-to-end Speech Recognition On Mobile Devices For Named Entitiescited by 41
  25. 2018PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognitioncited by 36
  26. 2023StyLIP: Multi-Scale Style-Conditioned Prompt Learning for CLIP-based Domain Generalizationcited by 36
  27. 2021Orienting Novel 3D Objects Using Self-Supervised Learning of Rotation Transformscited by 17
  28. 2020Domain-Specific Mappings for Generative Adversarial Style Transfercited by 16
  29. 20206-DoF Grasp Planning using Fast 3D Reconstruction and Grasp Quality CNNcited by 13
  30. 2022ART/ATK: A research platform for assessing and mitigating the sim-to-real gap in robotics and autonomous vehicle engineeringcited by 10
  31. 2022Language-guided Semantic Style Transfer of 3D Indoor Scenescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 14; 2025: 4; 2026: 1.

23 papers.

Sort:

By area:

  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  2. 2024Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectivescited by 27
  3. 2024DiffDet4SAR: Diffusion-based Aircraft Target Detection Network for SAR Imagescited by 15
  4. 2023MV-CLIP: Multi-View CLIP for Zero-shot 3D Shape Recognitioncited by 11
  5. 2023APLA: Additional Perturbation for Latent Noise with Adversarial Training Enables Consistencycited by 9
  6. 2024ShieldDiff: Suppressing Sexual Content Generation from Diffusion Models through Reinforcement Learningcited by 7
  7. 2024DEGSTalk: Decomposed Per-Embedding Gaussian Fields for Hair-Preserving Talking Face Synthesiscited by 7
  8. 2024CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimatorscited by 5
  9. 2023Open-Pose 3D Zero-Shot Learning: Benchmark and Challengescited by 4
  10. 2024PEVA-Net: Prompt-Enhanced View Aggregation Network for Zero/Few-Shot Multi-View 3D Shape Recognitioncited by 3
  11. 2024Learning Correspondence for Deformable Objectscited by 2
  12. 2024Safe Deep Model-Based Reinforcement Learning with Lyapunov Functionscited by 2
11 more
  1. 2024ED-SAM: An Efficient Diffusion Sampling Approach to Domain Generalization in Vision-Language Foundation Modelscited by 2
  2. 2024CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertaintycited by 2
  3. 2025HOMER: Homography-Based Efficient Multi-view 3D Object Removalcited by 2
  4. 2025H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flowscited by 1
  5. 2026CLIPoint3D: Language-Grounded Few-Shot Unsupervised 3D Point Cloud Domain Adaptationcited by 1
  6. 2024Towards Improving Learning from Demonstration Algorithms via MCMC Methodscited by 0
  7. 2024Theoretical Analysis for Expectation-Maximization-Based Multi-Model 3D Registrationcited by 0
  8. 2024Rocket Landing Control with Grid Fins and Path-following using MPCcited by 0
  9. 2024DC3DO: Diffusion Classifier for 3D Objectscited by 0
  10. 2025Towards Training-Free Open-World Classification with 3D Generative Modelscited by 0
  11. 2025NoiseShift: Resolution-Aware Noise Recalibration for Better Low-Resolution Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Diff-3DCap: Shape Captioning with Diffusion Models
  2. 2025Harnessing Text-to-Image Diffusion Models for Point Cloud Self-Supervised Learning
  3. 2024DiffCLIP: Few-shot Language-driven Multimodal Classifier
  4. 2024Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models
  5. 20243D Feature Distillation with Object-Centric Priors
  6. 2025DiffCLIP: Differential Attention Meets CLIP
  7. 2026How Do Diffusion Classifiers Decide? A Bias-Centric Evaluation
  8. 2023DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability