# Sparse-to-dense Feature Matching: Intra and Inter domain Cross-modal Learning in Domain Adaptation for 3D Semantic Segmentation

arXiv: https://arxiv.org/abs/2107.14724 · HTML: https://mlsanity.com/paper/2107.14724/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 51 references that are in the map.
- First posted on arXiv 30 Jul 2021.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds](https://mlsanity.com/paper/1911.11236/) (2019)
- [Learning to Adapt Structured Output Space for Semantic Segmentation](https://mlsanity.com/paper/1802.10349/) (2018)
- [Confidence Regularized Self-Training](https://mlsanity.com/paper/1908.09822/) (2019)
- [Generative Visual Manipulation on the Natural Image Manifold](https://mlsanity.com/paper/1609.03552/) (2016)
- [Unsupervised Domain Adaptation with Residual Transfer Networks](https://mlsanity.com/paper/1602.04433/) (2016)
- [Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks](https://mlsanity.com/paper/1612.05424/) (2016)
- [Bidirectional Learning for Domain Adaptation of Semantic Segmentation](https://mlsanity.com/paper/1904.10620/) (2019)
- [Taking A Closer Look at Domain Shift: Category-level Adversaries for Semantics Consistent Domain Adaptation](https://mlsanity.com/paper/1809.09478/) (2018)
- [FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation](https://mlsanity.com/paper/1612.02649/) (2016)
- [SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation for Road-Object Segmentation from a LiDAR Point Cloud](https://mlsanity.com/paper/1809.08495/) (2018)
- [Simultaneous Deep Transfer Across Domains and Tasks](https://mlsanity.com/paper/1510.02192/) (2015)
- [A2D2: Audi Autonomous Driving Dataset](https://mlsanity.com/paper/2004.06320/) (2020)
- [SPLATNet: Sparse Lattice Networks for Point Cloud Processing](https://mlsanity.com/paper/1802.08275/) (2018)
- [xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/1911.12676/) (2019)
- [No More Discrimination: Cross City Adaptation of Road Scene Segmenters](https://mlsanity.com/paper/1704.08509/) (2017)
- [Amulet: Aggregating Multi-level Convolutional Features for Salient Object Detection](https://mlsanity.com/paper/1708.02001/) (2017)

## Built on by (most cited first, 30 of 46)

- [Segment Any Point Cloud Sequences by Distilling Vision Foundation Models](https://mlsanity.com/paper/2306.09347/) (2023)
- [Semantic-Aware Domain Generalized Segmentation](https://mlsanity.com/paper/2204.00822/) (2022)
- [MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2204.12667/) (2022)
- [PiMAE: Point Cloud and Image Interactive Masked Autoencoders for 3D Object Detection](https://mlsanity.com/paper/2303.08129/) (2023)
- [Three Pillars improving Vision Foundation Model Distillation for Lidar](https://mlsanity.com/paper/2310.17504/) (2023)
- [ConDA: Unsupervised Domain Adaptation for LiDAR Segmentation via Regularized Domain Concatenation](https://mlsanity.com/paper/2111.15242/) (2021)
- [Deformation and Correspondence Aware Unsupervised Synthetic-to-Real Scene Flow Estimation for Point Clouds](https://mlsanity.com/paper/2203.16895/) (2022)
- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [SALUDA: Surface-based Automotive Lidar Unsupervised Domain Adaptation](https://mlsanity.com/paper/2304.03251/) (2023)
- [Multi-Modal Continual Test-Time Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2303.10457/) (2023)
- [A Survey of Label-Efficient Deep Learning for 3D Point Clouds](https://mlsanity.com/paper/2305.19812/) (2023)
- [Unsupervised Domain Adaptation via Domain-Adaptive Diffusion](https://mlsanity.com/paper/2308.13893/) (2023)
- [Learning to Adapt SAM for Segmenting Cross-domain Point Clouds](https://mlsanity.com/paper/2310.08820/) (2023)
- [MoPA: Multi-Modal Prior Aided Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2309.11839/) (2023)
- [MCD: Diverse Large-Scale Multi-Campus Dataset for Robot Perception](https://mlsanity.com/paper/2403.11496/) (2024)
- [Image Understands Point Cloud: Weakly Supervised 3D Semantic Segmentation via Association Learning](https://mlsanity.com/paper/2209.07774/) (2022)
- [Visual Foundation Models Boost Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2403.10001/) (2024)
- [Exploiting the Complementarity of 2D and 3D Networks to Address Domain-Shift in 3D Semantic Segmentation](https://mlsanity.com/paper/2304.02991/) (2023)
- [BEV-DG: Cross-Modal Learning under Bird's-Eye View for Domain Generalization of 3D Semantic Segmentation](https://mlsanity.com/paper/2308.06530/) (2023)
- [DODA: Data-oriented Sim-to-Real Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2204.01599/) (2022)
- [360^circ from a Single Camera: A Few-Shot Approach for LiDAR Segmentation](https://mlsanity.com/paper/2309.06197/) (2023)
- [Recent Advances in Multi-modal 3D Intelligence: A Comprehensive Survey and Evaluation](https://mlsanity.com/paper/2310.15676/) (2023)
- [LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving](https://mlsanity.com/paper/2501.04005/) (2025)
- [Decoupled Pseudo-labeling for Semi-Supervised Monocular 3D Object Detection](https://mlsanity.com/paper/2403.17387/) (2024)
- [Mx2M: Masked Cross-Modality Modeling in Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2307.04231/) (2023)
- [SCoDA: Domain Adaptive Shape Completion for Real Scans](https://mlsanity.com/paper/2304.10179/) (2023)
- [DualCross: Cross-Modality Cross-Domain Adaptation for Monocular BEV Perception](https://mlsanity.com/paper/2305.03724/) (2023)
- [Unsupervised Domain Adaptation for Point Cloud Semantic Segmentation via Graph Matching](https://mlsanity.com/paper/2208.04510/) (2022)
- [UFO: Uncertainty-aware LiDAR-image Fusion for Off-road Semantic Terrain Map Estimation](https://mlsanity.com/paper/2403.02642/) (2024)
- [TTT-KD: Test-Time Training for 3D Semantic Segmentation through Knowledge Distillation from Foundation Models](https://mlsanity.com/paper/2403.11691/) (2024)

## Abstract (arXiv)

Domain adaptation is critical for success when confronting with the lack of annotations in a new domain. As the huge time consumption of labeling process on 3D point cloud, domain adaptation for 3D semantic segmentation is of great expectation. With the rise of multi-modal datasets, large amount of 2D images are accessible besides 3D point clouds. In light of this, we propose to further leverage 2D data for 3D domain adaptation by intra and inter domain cross modal learning. As for intra-domain cross modal learning, most existing works sample the dense 2D pixel-wise features into the same size with sparse 3D point-wise features, resulting in the abandon of numerous useful 2D features. To address this problem, we propose Dynamic sparse-to-dense Cross Modal Learning (DsCML) to increase the sufficiency of multi-modality information interaction for domain adaptation. For inter-domain cross modal learning, we further advance Cross Modal Adversarial Learning (CMAL) on 2D and 3D data which contains different semantic content aiming to promote high-level modal complementarity. We evaluate our model under various multi-modality domain adaptation settings including day-to-night, country-to-country and dataset-to-dataset, brings large improvements over both uni-modal and multi-modal domain adaptation methods on all settings.