# Mix3D: Out-of-Context Data Augmentation for 3D Scenes

arXiv: https://arxiv.org/abs/2110.02210 · HTML: https://mlsanity.com/paper/2110.02210/

## Key facts

- Cited by 126 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 70 references that are in the map.
- First posted on arXiv 5 Oct 2021.

## Builds on (most cited first, 30 of 44)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Random Erasing Data Augmentation](https://mlsanity.com/paper/1708.04896/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [PointCNN: Convolution On X-Transformed Points](https://mlsanity.com/paper/1801.07791/) (2018)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup](https://mlsanity.com/paper/2009.06962/) (2020)
- [PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Sampling](https://mlsanity.com/paper/2003.00492/) (2020)
- [Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection](https://mlsanity.com/paper/1708.01642/) (2017)
- [SEGCloud: Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/1710.07563/) (2017)
- [OccuSeg: Occupancy-aware 3D Instance Segmentation](https://mlsanity.com/paper/2003.06537/) (2020)
- [Deep Parametric Continuous Convolutional Neural Networks](https://mlsanity.com/paper/2101.06742/) (2021)
- [MixUp as Locally Linear Out-Of-Manifold Regularization](https://mlsanity.com/paper/1809.02499/) (2018)
- [SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization](https://mlsanity.com/paper/2006.01791/) (2020)
- [3D-MPA: Multi Proposal Aggregation for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2003.13867/) (2020)
- [GridMask Data Augmentation](https://mlsanity.com/paper/2001.04086/) (2020)
- [Augmented Reality Meets Computer Vision : Efficient Data Generation for Urban Driving Scenes](https://mlsanity.com/paper/1708.01566/) (2017)
- [Modeling Visual Context is Key to Augmenting Object Detection Datasets](https://mlsanity.com/paper/1807.07428/) (2018)
- [Self-Supervised Learning for Domain Adaptation on Point-Clouds](https://mlsanity.com/paper/2003.12641/) (2020)
- [Virtual Multi-view Fusion for 3D Semantic Segmentation](https://mlsanity.com/paper/2007.13138/) (2020)
- [Bidirectional Projection Network for Cross Dimension Scene Understanding](https://mlsanity.com/paper/2103.14326/) (2021)

## Built on by (most cited first, 30 of 126)

- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)
- [OpenMask3D: Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2306.13631/) (2023)
- [Language-Grounded Indoor 3D Semantic Segmentation in the Wild](https://mlsanity.com/paper/2204.07761/) (2022)
- [Robo3D: Towards Robust and Reliable 3D Perception against Corruptions](https://mlsanity.com/paper/2303.17597/) (2023)
- [Rethinking Range View Representation for LiDAR Segmentation](https://mlsanity.com/paper/2303.05367/) (2023)
- [Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understanding](https://mlsanity.com/paper/2304.06906/) (2023)
- [OctFormer: Octree-based Transformers for 3D Point Clouds](https://mlsanity.com/paper/2305.03045/) (2023)
- [LaserMix for Semi-Supervised LiDAR Semantic Segmentation](https://mlsanity.com/paper/2207.00026/) (2022)
- [SAI3D: Segment Any Instance in 3D Scenes](https://mlsanity.com/paper/2312.11557/) (2023)
- [Compositional Human-Scene Interaction Synthesis with Semantic Control](https://mlsanity.com/paper/2207.12824/) (2022)
- [PolarMix: A General Data Augmentation Technique for LiDAR Point Clouds](https://mlsanity.com/paper/2208.00223/) (2022)
- [OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views](https://mlsanity.com/paper/2404.03650/) (2024)
- [Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labels](https://mlsanity.com/paper/2312.17232/) (2023)
- [UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebase](https://mlsanity.com/paper/2309.05573/) (2023)
- [Contrastive Boundary Learning for Point Cloud Segmentation](https://mlsanity.com/paper/2203.05272/) (2022)
- [Masked Scene Contrast: A Scalable Framework for Unsupervised 3D Representation Learning](https://mlsanity.com/paper/2303.14191/) (2023)
- [4DContrast: Contrastive Learning with Dynamic Correspondences for 3D Scene Understanding](https://mlsanity.com/paper/2112.02990/) (2021)
- [CoSMix: Compositional Semantic Mix for Domain Adaptation in 3D LiDAR Segmentation](https://mlsanity.com/paper/2207.09778/) (2022)
- [Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving](https://mlsanity.com/paper/2405.05258/) (2024)
- [FRNet: Frustum-Range Networks for Scalable LiDAR Segmentation](https://mlsanity.com/paper/2312.04484/) (2023)
- [Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation](https://mlsanity.com/paper/2204.07548/) (2022)
- [3D-Aware Object Goal Navigation via Simultaneous Exploration and Identification](https://mlsanity.com/paper/2212.00338/) (2022)
- [Understanding Imbalanced Semantic Segmentation Through Neural Collapse](https://mlsanity.com/paper/2301.01100/) (2023)
- [ConDA: Unsupervised Domain Adaptation for LiDAR Segmentation via Regularized Domain Concatenation](https://mlsanity.com/paper/2111.15242/) (2021)
- [GrowSP: Unsupervised Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2305.16404/) (2023)
- [DI-V2X: Learning Domain-Invariant Representation for Vehicle-Infrastructure Collaborative 3D Object Detection](https://mlsanity.com/paper/2312.15742/) (2023)
- [PointConvFormer: Revenge of the Point-based Convolution](https://mlsanity.com/paper/2208.02879/) (2022)
- [Divide and Conquer: 3D Point Cloud Instance Segmentation With Point-Wise Binarization](https://mlsanity.com/paper/2207.11209/) (2022)
- [Multi-Space Alignments Towards Universal LiDAR Segmentation](https://mlsanity.com/paper/2405.01538/) (2024)

## Abstract (arXiv)

We present Mix3D, a data augmentation technique for segmenting large-scale 3D scenes. Since scene context helps reasoning about object semantics, current works focus on models with large capacity and receptive fields that can fully capture the global context of an input 3D scene. However, strong contextual priors can have detrimental implications like mistaking a pedestrian crossing the street for a car. In this work, we focus on the importance of balancing global scene context and local geometry, with the goal of generalizing beyond the contextual priors in the training set. In particular, we propose a "mixing" technique which creates new training samples by combining two augmented scenes. By doing so, object instances are implicitly placed into novel out-of-context environments and therefore making it harder for models to rely on scene context alone, and instead infer semantics from local structure as well. We perform detailed analysis to understand the importance of global context, local structures and the effect of mixing scenes. In experiments, we show that models trained with Mix3D profit from a significant performance boost on indoor (ScanNet, S3DIS) and outdoor datasets (SemanticKITTI). Mix3D can be trivially used with any existing method, e.g., trained with Mix3D, MinkowskiNet outperforms all prior state-of-the-art methods by a significant margin on the ScanNet test benchmark 78.1 mIoU. Code is available at: https://nekrasov.dev/mix3d/