# Sculpting Holistic 3D Representation in Contrastive Language-Image-3D Pre-training

arXiv: https://arxiv.org/abs/2311.01734 · HTML: https://mlsanity.com/paper/2311.01734/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 76 of its 90 references that are in the map.
- First posted on arXiv 3 Nov 2023.

## Builds on (most cited first, 30 of 76)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation](https://mlsanity.com/paper/2205.13542/) (2022)
- [Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling](https://mlsanity.com/paper/2111.14819/) (2021)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)
- [Zero-Shot Text-Guided Object Generation with Dream Fields](https://mlsanity.com/paper/2112.01455/) (2021)
- [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020)
- [TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers](https://mlsanity.com/paper/2203.11496/) (2022)

## Built on by (most cited first, 18 of 18)

- [ShapeLLM: Universal 3D Object Understanding for Embodied Interaction](https://mlsanity.com/paper/2402.17766/) (2024)
- [Duoduo CLIP: Efficient 3D Understanding with Multi-View Images](https://mlsanity.com/paper/2406.11579/) (2024)
- [More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding](https://mlsanity.com/paper/2408.15966/) (2024)
- [DreamView: Injecting View-specific Text Guidance into Text-to-3D Generation](https://mlsanity.com/paper/2404.06119/) (2024)
- [Panoptic Captioning: An Equivalence Bridge for Image and Text](https://mlsanity.com/paper/2505.16334/) (2025)
- [A Survey on Self-supervised Contrastive Learning for Multimodal Text-Image Analysis](https://mlsanity.com/paper/2503.11101/) (2025)
- [Occlusion-aware Text-Image-Point Cloud Pretraining for Open-World 3D Object Recognition](https://mlsanity.com/paper/2502.10674/) (2025)
- [Point Cloud as a Foreign Language for Multi-modal Large Language Model](https://mlsanity.com/paper/2603.09173/) (2026)
- [Tango3D: Towards Alignment for Global and Local 2D-3D Correspondence](https://mlsanity.com/paper/2605.19727/) (2026)
- [TIGaussian: Disentangle Gaussians for Spatial-Awared Text-Image-3D Alignment](https://mlsanity.com/paper/2601.19247/) (2026)
- [BRepCLIP: Contrastive Multimodal Pretraining on BRep Primitives for CAD Understanding](https://mlsanity.com/paper/2606.05515/) (2026)
- [3D-MRL: Nested Multimodal 3D Representations via Matryoshka Representation Learning](https://mlsanity.com/paper/2608.29285/) (2026)
- [Text-guided Synthetic Geometric Augmentation for Zero-shot 3D Understanding](https://mlsanity.com/paper/2501.09278/) (2025)
- [SAMURAI: Shape-Aware Multimodal Retrieval for 3D Object Identification](https://mlsanity.com/paper/2506.21056/) (2025)
- [SCENEFORGE: Enhancing 3D-text alignment with Structured Scene Compositions](https://mlsanity.com/paper/2509.15693/) (2025)
- [HOLA: Holistic Multi-Modal Alignment for Open-Set 3D Recognition](https://mlsanity.com/paper/2606.01334/) (2026)
- [An overview of 3D Vision-Language Models](https://mlsanity.com/paper/2609.05583/) (2026)
- [Revisiting Triplet Contrastive Learning for 3D Shape Understanding](https://mlsanity.com/paper/2610.19519/) (2026)

## Abstract (arXiv)

Contrastive learning has emerged as a promising paradigm for 3D open-world understanding, i.e., aligning point cloud representation to image and text embedding space individually. In this paper, we introduce MixCon3D, a simple yet effective method aiming to sculpt holistic 3D representation in contrastive language-image-3D pre-training. In contrast to point cloud only, we develop the 3D object-level representation from complementary perspectives, e.g., multi-view rendered images with the point cloud. Then, MixCon3D performs language-3D contrastive learning, comprehensively depicting real-world 3D objects and bolstering text alignment. Additionally, we pioneer the first thorough investigation of various training recipes for the 3D contrastive learning paradigm, building a solid baseline with improved performance. Extensive experiments conducted on three representative benchmarks reveal that our method significantly improves over the baseline, surpassing the previous state-of-the-art performance on the challenging 1,156-category Objaverse-LVIS dataset by 5.7%. The versatility of MixCon3D is showcased in applications such as text-to-3D retrieval and point cloud captioning, further evidencing its efficacy in diverse scenarios. The code is available at https://github.com/UCSC-VLAA/MixCon3D.