# Cross-modal Learning for Domain Adaptation in 3D Semantic Segmentation

arXiv: https://arxiv.org/abs/2101.07253 · HTML: https://mlsanity.com/paper/2101.07253/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 62 references that are in the map.
- First posted on arXiv 18 Jan 2021.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds](https://mlsanity.com/paper/1911.11236/) (2019)
- [Learning to Adapt Structured Output Space for Semantic Segmentation](https://mlsanity.com/paper/1802.10349/) (2018)
- [FDA: Fourier Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2004.05498/) (2020)
- [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution](https://mlsanity.com/paper/2007.16100/) (2020)
- [Confidence Regularized Self-Training](https://mlsanity.com/paper/1908.09822/) (2019)
- [Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation](https://mlsanity.com/paper/2011.10033/) (2020)
- [OctNet: Learning Deep 3D Representations at High Resolutions](https://mlsanity.com/paper/1611.05009/) (2016)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [Semi-supervised Domain Adaptation via Minimax Entropy](https://mlsanity.com/paper/1904.06487/) (2019)
- [Bidirectional Learning for Domain Adaptation of Semantic Segmentation](https://mlsanity.com/paper/1904.10620/) (2019)
- [PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation](https://mlsanity.com/paper/2003.14032/) (2020)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [Deep Continuous Fusion for Multi-Sensor 3D Object Detection](https://mlsanity.com/paper/2012.10992/) (2020)
- [FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation](https://mlsanity.com/paper/1612.02649/) (2016)
- [SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud](https://mlsanity.com/paper/1710.07368/) (2017)
- [SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation for Road-Object Segmentation from a LiDAR Point Cloud](https://mlsanity.com/paper/1809.08495/) (2018)

## Built on by (most cited first, 30 of 40)

- [Segment Any Point Cloud Sequences by Distilling Vision Foundation Models](https://mlsanity.com/paper/2306.09347/) (2023)
- [Learning 3D Semantic Segmentation with only 2D Image Supervision](https://mlsanity.com/paper/2110.11325/) (2021)
- [Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving](https://mlsanity.com/paper/2405.05258/) (2024)
- [PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection](https://mlsanity.com/paper/2303.08129/) (2023)
- [FRNet: Frustum-Range Networks for Scalable LiDAR Segmentation](https://mlsanity.com/paper/2312.04484/) (2023)
- [Synthetic Datasets for Autonomous Driving: A Survey](https://mlsanity.com/paper/2304.12205/) (2023)
- [Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Driving](https://mlsanity.com/paper/2405.17426/) (2024)
- [Multi-Space Alignments Towards Universal LiDAR Segmentation](https://mlsanity.com/paper/2405.01538/) (2024)
- [SALUDA: Surface-based Automotive Lidar Unsupervised Domain Adaptation](https://mlsanity.com/paper/2304.03251/) (2023)
- [A Survey of Label-Efficient Deep Learning for 3D Point Clouds](https://mlsanity.com/paper/2305.19812/) (2023)
- [Compositional Semantic Mix for Domain Adaptation in Point Cloud Segmentation](https://mlsanity.com/paper/2308.14619/) (2023)
- [Learning to Adapt SAM for Segmenting Cross-domain Point Clouds](https://mlsanity.com/paper/2310.08820/) (2023)
- [Visual Foundation Models Boost Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2403.10001/) (2024)
- [Exploiting the Complementarity of 2D and 3D Networks to Address Domain-Shift in 3D Semantic Segmentation](https://mlsanity.com/paper/2304.02991/) (2023)
- [Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluation](https://mlsanity.com/paper/2310.15676/) (2023)
- [LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving](https://mlsanity.com/paper/2501.04005/) (2025)
- [SPIRAL: Semantic-Aware Progressive LiDAR Scene Generation and Understanding](https://mlsanity.com/paper/2505.22643/) (2025)
- [Advancing 3D Point Cloud Understanding through Deep Transfer Learning: A Comprehensive Survey](https://mlsanity.com/paper/2407.17877/) (2024)
- [Source-Free Domain Adaptation for RGB-D Semantic Segmentation with Vision Transformers](https://mlsanity.com/paper/2305.14269/) (2023)
- [Understanding Dark Scenes by Contrasting Multi-Modal Observations](https://mlsanity.com/paper/2308.12320/) (2023)
- [LiOn-XA: Unsupervised Domain Adaptation via LiDAR-Only Cross-Modal Adversarial Training](https://mlsanity.com/paper/2410.15833/) (2024)
- [Fusion-then-Distillation: Toward Cross-modal Positive Distillation for Domain Adaptive 3D Semantic Segmentation](https://mlsanity.com/paper/2410.19446/) (2024)
- [A Review of 3D Object Detection with Vision-Language Models](https://mlsanity.com/paper/2504.18738/) (2025)
- [Model2Scene: Learning 3D Scene Representation via Contrastive Language-CAD Models Pre-training](https://mlsanity.com/paper/2309.16956/) (2023)
- [Heterogeneous-Modal Unsupervised Domain Adaptation via Latent Space Bridging](https://mlsanity.com/paper/2506.15971/) (2025)
- [Domain-aware Category-level Geometry Learning Segmentation for 3D Point Clouds](https://mlsanity.com/paper/2508.11265/) (2025)
- [MOCHA: Multi-modal Objects-aware Cross-arcHitecture Alignment](https://mlsanity.com/paper/2509.14001/) (2025)
- [Improving Multimodal Distillation for 3D Semantic Segmentation under Domain Shift](https://mlsanity.com/paper/2511.17455/) (2025)
- [UniDA3D: Unified Domain Adaptive 3D Semantic Segmentation Pipeline](https://mlsanity.com/paper/2212.10390/) (2022)
- [Zero-shot Learning of Drug Response Prediction for Preclinical Drug Screening](https://mlsanity.com/paper/2310.12996/) (2023)

## Abstract (arXiv)

Domain adaptation is an important task to enable learning when labels are scarce. While most works focus only on the image modality, there are many important multi-modal datasets. In order to leverage multi-modality for domain adaptation, we propose cross-modal learning, where we enforce consistency between the predictions of two modalities via mutual mimicking. We constrain our network to make correct predictions on labeled data and consistent predictions across modalities on unlabeled target-domain data. Experiments in unsupervised and semi-supervised domain adaptation settings prove the effectiveness of this novel domain adaptation strategy. Specifically, we evaluate on the task of 3D semantic segmentation from either the 2D image, the 3D point cloud or from both. We leverage recent driving datasets to produce a wide variety of domain adaptation scenarios including changes in scene layout, lighting, sensor setup and weather, as well as the synthetic-to-real setup. Our method significantly improves over previous uni-modal adaptation baselines on all adaption scenarios. Our code is publicly available at https://github.com/valeoai/xmuda_journal