# No Captions, No Problem: Captionless 3D-CLIP Alignment with Hard Negatives via CLIP Knowledge and LLMs

arXiv: https://arxiv.org/abs/2406.02202 · HTML: https://mlsanity.com/paper/2406.02202/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 28 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)
- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding](https://mlsanity.com/paper/2212.05171/) (2022)
- [ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding](https://mlsanity.com/paper/2305.08275/) (2023)
- [PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning](https://mlsanity.com/paper/2211.11682/) (2022)
- [Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining](https://mlsanity.com/paper/2302.02318/) (2023)
- [CLIP2Point: Transfer CLIP to Point Cloud Classification with Image-Depth Pre-training](https://mlsanity.com/paper/2210.01055/) (2022)
- [Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training](https://mlsanity.com/paper/2301.02280/) (2023)
- [SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger](https://mlsanity.com/paper/2303.17561/) (2023)
- [Comprehensive Review of Deep Learning-Based 3D Point Cloud Completion Processing and Analysis](https://mlsanity.com/paper/2203.03311/) (2022)
- [IC3D: Image-Conditioned 3D Diffusion for Shape Generation](https://mlsanity.com/paper/2211.10865/) (2022)

## Built on by (most cited first, 1 of 1)

- [On the Limitations of Vision-Language Models in Understanding Image Transforms](https://mlsanity.com/paper/2503.09837/) (2025)

## Abstract (arXiv)

In this study, we explore an alternative approach to enhance contrastive text-image-3D alignment in the absence of textual descriptions for 3D objects. We introduce two unsupervised methods, I2I and (I2L)2, which leverage CLIP knowledge about textual and 2D data to compute the neural perceived similarity between two 3D samples. We employ the proposed methods to mine 3D hard negatives, establishing a multimodal contrastive pipeline with hard negative weighting via a custom loss function. We train on different configurations of the proposed hard negative mining approach, and we evaluate the accuracy of our models in 3D classification and on the cross-modal retrieval benchmark, testing image-to-shape and shape-to-image retrieval. Results demonstrate that our approach, even without explicit text alignment, achieves comparable or superior performance on zero-shot and standard 3D classification, while significantly improving both image-to-shape and shape-to-image retrieval compared to previous methods.