# Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretraining

arXiv: https://arxiv.org/abs/2302.02318 · HTML: https://mlsanity.com/paper/2302.02318/

## Key facts

- Cited by 164 papers in the map (data through 14 Oct 2026).
- Builds on 82 of its 107 references that are in the map.
- First posted on arXiv 5 Feb 2023.

## Builds on (most cited first, 30 of 82)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)

## Built on by (most cited first, 30 of 164)

- [DreamLLM: Synergistic Multimodal Comprehension and Creation](https://mlsanity.com/paper/2309.11499/) (2023)
- [Uni3D: Exploring Unified 3D Representation at Scale](https://mlsanity.com/paper/2310.06773/) (2023)
- [DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge](https://mlsanity.com/paper/2507.04447/) (2025)
- [ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding](https://mlsanity.com/paper/2305.08275/) (2023)
- [Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following](https://mlsanity.com/paper/2309.00615/) (2023)
- [OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understanding](https://mlsanity.com/paper/2305.10764/) (2023)
- [PointMamba: A Simple State Space Model for Point Cloud Analysis](https://mlsanity.com/paper/2402.10739/) (2024)
- [ShapeLLM: Universal 3D Object Understanding for Embodied Interaction](https://mlsanity.com/paper/2402.17766/) (2024)
- [PointGPT: Auto-regressively Generative Pre-training from Point Clouds](https://mlsanity.com/paper/2305.11487/) (2023)
- [OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Models](https://mlsanity.com/paper/2506.03135/) (2025)
- [CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP](https://mlsanity.com/paper/2303.04748/) (2023)
- [Instance-aware Dynamic Prompt Tuning for Pre-trained Point Cloud Models](https://mlsanity.com/paper/2304.07221/) (2023)
- [Mamba3D: Enhancing Local Features for 3D Point Cloud Analysis via State Space Model](https://mlsanity.com/paper/2404.14966/) (2024)
- [Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis](https://mlsanity.com/paper/2403.01439/) (2024)
- [SceneSplat: Gaussian Splatting-based Scene Understanding with Vision-Language Pretraining](https://mlsanity.com/paper/2503.18052/) (2025)
- [SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation](https://mlsanity.com/paper/2502.13143/) (2025)
- [OmniVec: Learning robust representations with cross modal sharing](https://mlsanity.com/paper/2311.05709/) (2023)
- [Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learning](https://mlsanity.com/paper/2410.08114/) (2024)
- [Towards Compact 3D Representations via Point Feature Enhancement Masked Autoencoders](https://mlsanity.com/paper/2312.10726/) (2023)
- [SGDrive: Scene-to-Goal Hierarchical World Cognition for Autonomous Driving](https://mlsanity.com/paper/2601.05640/) (2026)
- [Connecting Multi-modal Contrastive Representations](https://mlsanity.com/paper/2305.14381/) (2023)
- [TAMM: TriAdapter Multi-Modal Learning for 3D Shape Understanding](https://mlsanity.com/paper/2402.18490/) (2024)
- [SUGAR: Pre-training 3D Visual Representations for Robotics](https://mlsanity.com/paper/2404.01491/) (2024)
- [Take-A-Photo: 3D-to-2D Generative Pre-training of Point Cloud Models](https://mlsanity.com/paper/2307.14971/) (2023)
- [PCP-MAE: Learning to Predict Centers for Point Masked Autoencoders](https://mlsanity.com/paper/2408.08753/) (2024)
- [DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classification](https://mlsanity.com/paper/2305.15957/) (2023)
- [Any2Point: Empowering Any-modality Large Models for Efficient 3D Understanding](https://mlsanity.com/paper/2404.07989/) (2024)
- [DropMAE: Learning Representations via Masked Autoencoders with Spatial-Attention Dropout for Temporal Matching Tasks](https://mlsanity.com/paper/2304.00571/) (2023)
- [PointRWKV: Efficient RWKV-Like Model for Hierarchical Point Cloud Learning](https://mlsanity.com/paper/2405.15214/) (2024)
- [ShapeSplat: A Large-scale Dataset of Gaussian Splats and Their Self-Supervised Pretraining](https://mlsanity.com/paper/2408.10906/) (2024)

## Abstract (arXiv)

Mainstream 3D representation learning approaches are built upon contrastive or generative modeling pretext tasks, where great improvements in performance on various downstream tasks have been achieved. However, we find these two paradigms have different characteristics: (i) contrastive models are data-hungry that suffer from a representation over-fitting issue; (ii) generative models have a data filling issue that shows inferior data scaling capacity compared to contrastive models. This motivates us to learn 3D representations by sharing the merits of both paradigms, which is non-trivial due to the pattern difference between the two paradigms. In this paper, we propose Contrast with Reconstruct (ReCon) that unifies these two paradigms. ReCon is trained to learn from both generative modeling teachers and single/cross-modal contrastive teachers through ensemble distillation, where the generative student guides the contrastive student. An encoder-decoder style ReCon-block is proposed that transfers knowledge through cross attention with stop-gradient, which avoids pretraining over-fitting and pattern difference issues. ReCon achieves a new state-of-the-art in 3D representation learning, e.g., 91.26% accuracy on ScanObjectNN. Codes have been released at https://github.com/qizekun/ReCon.