# RandomRooms: Unsupervised Pre-training from Synthetic Shapes and Randomized Layouts for 3D Object Detection

arXiv: https://arxiv.org/abs/2108.07794 · HTML: https://mlsanity.com/paper/2108.07794/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 66 references that are in the map.
- First posted on arXiv 17 Aug 2021.

## Builds on (most cited first, 30 of 50)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding](https://mlsanity.com/paper/2007.10985/) (2020)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)

## Built on by (most cited first, 30 of 37)

- [Language-Grounded Indoor 3D Semantic Segmentation in the Wild](https://mlsanity.com/paper/2204.07761/) (2022)
- [Masked Discrimination for Self-Supervised Learning on Point Clouds](https://mlsanity.com/paper/2203.11183/) (2022)
- [Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?](https://mlsanity.com/paper/2212.08320/) (2022)
- [Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understanding](https://mlsanity.com/paper/2304.06906/) (2023)
- [PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm](https://mlsanity.com/paper/2310.08586/) (2023)
- [4DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understanding](https://mlsanity.com/paper/2112.02990/) (2021)
- [Unsupervised Point Cloud Representation Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2202.13589/) (2022)
- [Ponder: Point Cloud Pre-training via Neural Rendering](https://mlsanity.com/paper/2301.00157/) (2022)
- [Implicit Autoencoder for Point-Cloud Self-Supervised Representation Learning](https://mlsanity.com/paper/2201.00785/) (2022)
- [ProposalContrast: Unsupervised Pre-training for LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2207.12654/) (2022)
- [Complete-to-Partial 4D Distillation for Self-Supervised Point Cloud Sequence Representation Learning](https://mlsanity.com/paper/2212.05330/) (2022)
- [3D Object Detection with a Self-supervised Lidar Scene Flow Backbone](https://mlsanity.com/paper/2205.00705/) (2022)
- [Back to Reality: Weakly-supervised 3D Object Detection with Shape-guided Label Enhancement](https://mlsanity.com/paper/2203.05238/) (2022)
- [Joint Representation Learning for Text and 3D Point Cloud](https://mlsanity.com/paper/2301.07584/) (2023)
- [SRFeat: Learning Locally Accurate and Globally Consistent Non-Rigid Shape Correspondence](https://mlsanity.com/paper/2209.07806/) (2022)
- [Self-supervised Pre-training with Masked Shape Prediction for 3D Scene Understanding](https://mlsanity.com/paper/2305.05026/) (2023)
- [MM-3DScene: 3D Scene Understanding by Customizing Masked Modeling with Informative-Preserved Reconstruction and Self-Distilled Consistency](https://mlsanity.com/paper/2212.09948/) (2022)
- [T-MAE: Temporal Masked Autoencoders for Point Cloud Representation Learning](https://mlsanity.com/paper/2312.10217/) (2023)
- [BSNet: Box-Supervised Simulation-assisted Mean Teacher for 3D Instance Segmentation](https://mlsanity.com/paper/2403.15019/) (2024)
- [3D Point Cloud Pre-training with Knowledge Distillation from 2D Images](https://mlsanity.com/paper/2212.08974/) (2022)
- [PRED: Pre-training via Semantic Rendering on LiDAR Point Clouds](https://mlsanity.com/paper/2311.04501/) (2023)
- [Hardness-Aware Scene Synthesis for Semi-Supervised 3D Object Detection](https://mlsanity.com/paper/2405.17422/) (2024)
- [UniPre3D: Unified Pre-training of 3D Point Cloud Models with Cross-Modal Gaussian Splatting](https://mlsanity.com/paper/2506.09952/) (2025)
- [PointPatchMix: Point Cloud Mixing with Patch Scoring](https://mlsanity.com/paper/2303.06678/) (2023)
- [MaskLRF: Self-supervised Pretraining via Masked Autoencoding of Local Reference Frames for Rotation-invariant 3D Point Set Analysis](https://mlsanity.com/paper/2403.00206/) (2024)
- [Multi-View Representation is What You Need for Point-Cloud Pre-Training](https://mlsanity.com/paper/2306.02558/) (2023)
- [HUNTER: Unsupervised Human-centric 3D Detection via Transferring Knowledge from Synthetic Instances to Real Scenes](https://mlsanity.com/paper/2403.02769/) (2024)
- [MM-Mixing: Multi-Modal Mixing Alignment for 3D Understanding](https://mlsanity.com/paper/2405.18523/) (2024)
- [Token Adaptation via Side Graph Convolution for Efficient Fine-tuning of 3D Point Cloud Transformers](https://mlsanity.com/paper/2502.14142/) (2025)
- [To Supervise or Not to Supervise: Understanding and Addressing the Key Challenges of Point Cloud Transfer Learning](https://mlsanity.com/paper/2403.17869/) (2024)

## Abstract (arXiv)

3D point cloud understanding has made great progress in recent years. However, one major bottleneck is the scarcity of annotated real datasets, especially compared to 2D object detection tasks, since a large amount of labor is involved in annotating the real scans of a scene. A promising solution to this problem is to make better use of the synthetic dataset, which consists of CAD object models, to boost the learning on real datasets. This can be achieved by the pre-training and fine-tuning procedure. However, recent work on 3D pre-training exhibits failure when transfer features learned on synthetic objects to other real-world applications. In this work, we put forward a new method called RandomRooms to accomplish this objective. In particular, we propose to generate random layouts of a scene by making use of the objects in the synthetic CAD dataset and learn the 3D scene representation by applying object-level contrastive learning on two random scenes generated from the same set of synthetic objects. The model pre-trained in this way can serve as a better initialization when later fine-tuning on the 3D object detection task. Empirically, we show consistent improvement in downstream 3D detection tasks on several base models, especially when less training data are used, which strongly demonstrates the effectiveness and generalization of our method. Benefiting from the rich semantic knowledge and diverse objects from synthetic data, our method establishes the new state-of-the-art on widely-used 3D detection benchmarks ScanNetV2 and SUN RGB-D. We expect our attempt to provide a new perspective for bridging object and scene-level 3D understanding.