mlsanityData to 7 Oct 2026

Learning Robust 3D Representation from CLIP via Dual Denoising

arXiv 2407.00905 ↗First posted 1 Jul 2024cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

In this paper, we explore a critical yet under-investigated issue: how to learn robust and well-generalized 3D representation from pre-trained vision language models such as CLIP. Previous works have demonstrated that cross-modal distillation can provide rich and useful knowledge for 3D data. However, like most deep learning models, the resultant 3D learning network is still vulnerable to adversarial attacks especially the iterative attack. In this work, we propose Dual Denoising, a novel framework for learning robust and well-generalized 3D representations from CLIP. It combines a denoising-based proxy task with a novel feature denoising network for 3D pre-training. Additionally, we propose utilizing parallel noise inference to enhance the generalization of point cloud features under cross domain settings. Experiments show that our model can effectively improve the representation learning performance and adversarial robustness of the 3D learning network under zero-shot settings without adversarial training. Our code is available at https://github.com/luoshuqing2001/Dual_Denoising.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 47 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

38 of 47 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2022Scalable Diffusion Models with Transformerscited by 5,931
  10. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  11. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
26 more
  1. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  4. 2020Point Transformercited by 1,462
  5. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  6. 2020PCT: Point cloud transformercited by 730
  7. 2019Adversarial Training for Free!cited by 638
  8. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  9. 2022Diffusion Models for Adversarial Purificationcited by 469
  10. 2021PointCLIP: Point Cloud Understanding by CLIPcited by 461
  11. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 434
  12. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  13. 2018Feature Denoising for Improving Adversarial Robustnesscited by 312
  14. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 262
  15. 2023Uni3D: Exploring Unified 3D Representation at Scalecited by 231
  16. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  17. 2022PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learningcited by 186
  18. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  19. 2022CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-trainingcited by 159
  20. 2018Generating 3D Adversarial Point Cloudscited by 144
  21. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  22. 2021Adversarial purification with Score-based generative modelscited by 132
  23. 2023Denoising Diffusion Autoencoders are Unified Self-supervised Learnerscited by 128
  24. 2024Deconstructing Denoising Diffusion Models for Self-Supervised Learningcited by 89
  25. 2019AdvPC: Transferable Adversarial Perturbations on 3D Point Cloudscited by 84
  26. 2023CLIP2: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Datacited by 69

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Positional Prompt Tuning for Efficient 3D Representation Learningcited by 15

Similar, not cited

Close in wording, no citation either way

  1. 2022Ada3Diff: Defending against 3D Adversarial Point Clouds via Adaptive Diffusion
  2. 2025CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification
  3. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models
  4. 2024Few-shot point cloud reconstruction and denoising via learned Guassian splats renderings and fine-tuned diffusion features
  5. 2026CLIPoint3D: Language-Grounded Few-Shot Unsupervised 3D Point Cloud Domain Adaptation
  6. 2025Parameter-Efficient CLIP Adaptation for 3D Understanding via Unified Tokenization
  7. 2024GS-CLIP: Gaussian Splatting for Contrastive Language-Image-3D Pretraining from Real-World Data
  8. 2024An End-to-End Robust Point Cloud Semantic Segmentation Network with Single-Step Conditional Diffusion Models