# Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learning

arXiv: https://arxiv.org/abs/2303.14191 · HTML: https://mlsanity.com/paper/2303.14191/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 63 references that are in the map.
- First posted on arXiv 24 Mar 2023.

## Builds on (most cited first, 30 of 45)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Masked Feature Prediction for Self-Supervised Visual Pre-Training](https://mlsanity.com/paper/2112.09133/) (2021)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [Point Transformer V2: Grouped Vector Attention and Partition-based Pooling](https://mlsanity.com/paper/2210.05666/) (2022)
- [Deep Closest Point: Learning Representations for Point Cloud Registration](https://mlsanity.com/paper/1905.03304/) (2019)

## Built on by (most cited first, 30 of 56)

- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understanding](https://mlsanity.com/paper/2304.06906/) (2023)
- [Sonata: Self-Supervised Learning of Reliable Point Representations](https://mlsanity.com/paper/2503.16429/) (2025)
- [Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training](https://mlsanity.com/paper/2308.09718/) (2023)
- [UniPAD: A Universal Pre-training Paradigm for Autonomous Driving](https://mlsanity.com/paper/2310.08370/) (2023)
- [Point Cloud Mamba: Point Cloud Learning via State Space Model](https://mlsanity.com/paper/2403.00762/) (2024)
- [PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm](https://mlsanity.com/paper/2310.08586/) (2023)
- [Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding](https://mlsanity.com/paper/2409.03757/) (2024)
- [Point Cloud Pre-training with Diffusion Models](https://mlsanity.com/paper/2311.14960/) (2023)
- [Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning](https://mlsanity.com/paper/2402.02500/) (2024)
- [Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations](https://mlsanity.com/paper/2510.23607/) (2025)
- [OA-CNNs: Omni-Adaptive Sparse CNNs for 3D Semantic Segmentation](https://mlsanity.com/paper/2403.14418/) (2024)
- [DINO in the Room: Leveraging 2D Foundation Models for 3D Segmentation](https://mlsanity.com/paper/2503.18944/) (2025)
- [Masked Image Modeling: A Survey](https://mlsanity.com/paper/2408.06687/) (2024)
- [GroupContrast: Semantic-aware Self-supervised Representation Learning for 3D Understanding](https://mlsanity.com/paper/2403.09639/) (2024)
- [Utonia: Toward One Encoder for All Point Clouds](https://mlsanity.com/paper/2603.03283/) (2026)
- [Bringing Masked Autoencoders Explicit Contrastive Properties for Point Cloud Self-Supervised Learning](https://mlsanity.com/paper/2407.05862/) (2024)
- [Point-GCC: Universal Self-supervised 3D Scene Pre-training via Geometry-Color Contrast](https://mlsanity.com/paper/2305.19623/) (2023)
- [LCM: Locally Constrained Compact Point Cloud Model for Masked Point Modeling](https://mlsanity.com/paper/2405.17149/) (2024)
- [Efficient 3D Recognition with Event-driven Spike Sparse Convolution](https://mlsanity.com/paper/2412.07360/) (2024)
- [Masked Point-Entity Contrast for Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2504.19500/) (2025)
- [Chorus: Multi-Teacher Pretraining for Holistic 3D Gaussian Scene Encoding](https://mlsanity.com/paper/2512.17817/) (2025)
- [ConDense: Consistent 2D/3D Pre-training for Dense and Sparse Features from Multi-View Images](https://mlsanity.com/paper/2408.17027/) (2024)
- [Swin3D++: Effective Multi-Source Pretraining for 3D Indoor Scene Understanding](https://mlsanity.com/paper/2402.14215/) (2024)
- [UniPre3D: Unified Pre-training of 3D Point Cloud Models with Cross-Modal Gaussian Splatting](https://mlsanity.com/paper/2506.09952/) (2025)
- [Mitigating Object Dependencies: Improving Point Cloud Self-Supervised Learning through Object Exchange](https://mlsanity.com/paper/2404.07504/) (2024)
- [Masked Scene Modeling: Narrowing the Gap Between Supervised and Self-Supervised Learning in 3D Scene Understanding](https://mlsanity.com/paper/2504.06719/) (2025)
- [On Geometry-Enhanced Parameter-Efficient Fine-Tuning for 3D Scene Segmentation](https://mlsanity.com/paper/2505.22444/) (2025)
- [Towards Foundation Models for 3D Scene Understanding: Instance-Aware Self-Supervised Learning for Point Clouds](https://mlsanity.com/paper/2603.25165/) (2026)
- [CleverDistiller: Simple and Spatially Consistent Cross-modal Distillation](https://mlsanity.com/paper/2503.09878/) (2025)

## Abstract (arXiv)

As a pioneering work, PointContrast conducts unsupervised 3D representation learning via leveraging contrastive learning over raw RGB-D frames and proves its effectiveness on various downstream tasks. However, the trend of large-scale unsupervised learning in 3D has yet to emerge due to two stumbling blocks: the inefficiency of matching RGB-D frames as contrastive views and the annoying mode collapse phenomenon mentioned in previous works. Turning the two stumbling blocks into empirical stepping stones, we first propose an efficient and effective contrastive learning framework, which generates contrastive views directly on scene-level point clouds by a well-curated data augmentation pipeline and a practical view mixing strategy. Second, we introduce reconstructive learning on the contrastive learning framework with an exquisite design of contrastive cross masks, which targets the reconstruction of point color and surfel normal. Our Masked Scene Contrast (MSC) framework is capable of extracting comprehensive 3D representations more efficiently and effectively. It accelerates the pre-training procedure by at least 3x and still achieves an uncompromised performance compared with previous work. Besides, MSC also enables large-scale 3D pre-training across multiple datasets, which further boosts the performance and achieves state-of-the-art fine-tuning results on several downstream tasks, e.g., 75.5% mIoU on ScanNet semantic segmentation validation set.