# MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation

arXiv: https://arxiv.org/abs/2204.12667 · HTML: https://mlsanity.com/paper/2204.12667/

## Key facts

- Cited by 76 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 27 Apr 2022.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation](https://mlsanity.com/paper/1811.12833/) (2018)
- [Learning to Adapt Structured Output Space for Semantic Segmentation](https://mlsanity.com/paper/1802.10349/) (2018)
- [Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution](https://mlsanity.com/paper/2007.16100/) (2020)
- [Confidence Regularized Self-Training](https://mlsanity.com/paper/1908.09822/) (2019)
- [SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud](https://mlsanity.com/paper/1710.07368/) (2017)
- [SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation for Road-Object Segmentation from a LiDAR Point Cloud](https://mlsanity.com/paper/1809.08495/) (2018)
- [Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1706.01307/) (2017)
- [A2D2: Audi Autonomous Driving Dataset](https://mlsanity.com/paper/2004.06320/) (2020)
- [RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation](https://mlsanity.com/paper/2103.12978/) (2021)
- [xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/1911.12676/) (2019)
- [Complete & Label: A Domain Adaptation Approach to Semantic Segmentation of LiDAR Point Clouds](https://mlsanity.com/paper/2007.08488/) (2020)
- [Two-phase Pseudo Label Densification for Self-training based Domain Adaptation](https://mlsanity.com/paper/2012.04828/) (2020)
- [Sensor Fusion for Joint 3D Object Detection and Semantic Segmentation](https://mlsanity.com/paper/1904.11466/) (2019)
- [Learning Cross-modal Contrastive Features for Video Domain Adaptation](https://mlsanity.com/paper/2108.11974/) (2021)
- [Sparse-to-dense Feature Matching: Intra and Inter domain Cross-modal Learning in Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2107.14724/) (2021)
- [RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/1906.00208/) (2019)
- [Domain Adaptation for Vehicle Detection from Bird's Eye View LiDAR Point Cloud Data](https://mlsanity.com/paper/1905.08955/) (2019)
- [Test-Time Personalization with a Transformer for Human Pose Estimation](https://mlsanity.com/paper/2107.02133/) (2021)
- [Domain Adaptation for Semantic Segmentation via Patch-Wise Contrastive Learning](https://mlsanity.com/paper/2104.11056/) (2021)
- [PointSeg: Real-Time Semantic Segmentation Based on 3D LiDAR Point Cloud](https://mlsanity.com/paper/1807.06288/) (2018)

## Built on by (most cited first, 30 of 76)

- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [Robust Test-Time Adaptation in Dynamic Scenarios](https://mlsanity.com/paper/2303.13899/) (2023)
- [EcoTTA: Memory-Efficient Continual Test-time Adaptation via Self-distilled Regularization](https://mlsanity.com/paper/2303.01904/) (2023)
- [A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trends](https://mlsanity.com/paper/2410.15067/) (2024)
- [Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models](https://mlsanity.com/paper/2410.12790/) (2024)
- [GrowSP: Unsupervised Semantic Segmentation of 3D Point Clouds](https://mlsanity.com/paper/2305.16404/) (2023)
- [TTA-COPE: Test-Time Adaptation for Category-Level Object Pose Estimation](https://mlsanity.com/paper/2303.16730/) (2023)
- [Exploring Sparse Visual Prompt for Domain Adaptive Dense Prediction](https://mlsanity.com/paper/2303.09792/) (2023)
- [Test-Time Adaptation for Depth Completion](https://mlsanity.com/paper/2402.03312/) (2024)
- [MATE: Masked Autoencoders are Online 3D Test-Time Learners](https://mlsanity.com/paper/2211.11432/) (2022)
- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [Multi-Modal Continual Test-Time Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2303.10457/) (2023)
- [A Survey of Label-Efficient Deep Learning for 3D Point Clouds](https://mlsanity.com/paper/2305.19812/) (2023)
- [Benchmarking Test-Time Adaptation against Distribution Shifts in Image Classification](https://mlsanity.com/paper/2307.03133/) (2023)
- [Test-Time Degradation Adaptation for Open-Set Image Restoration](https://mlsanity.com/paper/2312.02197/) (2023)
- [PatchRefiner: Leveraging Synthetic Data for Real-Domain High-Resolution Monocular Metric Depth Estimation](https://mlsanity.com/paper/2406.06679/) (2024)
- [Effective Restoration of Source Knowledge in Continual Test Time Adaptation](https://mlsanity.com/paper/2311.04991/) (2023)
- [Visual Foundation Models Boost Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2403.10001/) (2024)
- [Exploiting the Complementarity of 2D and 3D Networks to Address Domain-Shift in 3D Semantic Segmentation](https://mlsanity.com/paper/2304.02991/) (2023)
- [Backpropagation-free Network for 3D Test-time Adaptation](https://mlsanity.com/paper/2403.18442/) (2024)
- [TALoS: Enhancing Semantic Scene Completion via Test-time Adaptation on the Line of Sight](https://mlsanity.com/paper/2410.15674/) (2024)
- [Towards Robust Multimodal Open-set Test-time Adaptation via Adaptive Entropy-aware Optimization](https://mlsanity.com/paper/2501.13924/) (2025)
- [Annotator: A Generic Active Learning Baseline for LiDAR Semantic Segmentation](https://mlsanity.com/paper/2310.20293/) (2023)
- [Beyond Entropy: Region Confidence Proxy for Wild Test-Time Adaptation](https://mlsanity.com/paper/2505.20704/) (2025)
- [Deep Transfer Learning for Intelligent Vehicle Perception: a Survey](https://mlsanity.com/paper/2306.15110/) (2023)
- [Distribution-Aware Continual Test-Time Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2309.13604/) (2023)
- [Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noises](https://mlsanity.com/paper/2503.02616/) (2025)
- [Enhancing and Adapting in the Clinic: Source-free Unsupervised Domain Adaptation for Medical Image Enhancement](https://mlsanity.com/paper/2312.01338/) (2023)
- [Source-Free Domain Adaptation for RGB-D Semantic Segmentation with Vision Transformers](https://mlsanity.com/paper/2305.14269/) (2023)
- [GeoAdapt: Self-Supervised Test-Time Adaptation in LiDAR Place Recognition Using Geometric Priors](https://mlsanity.com/paper/2308.04638/) (2023)

## Abstract (arXiv)

Test-time adaptation approaches have recently emerged as a practical solution for handling domain shift without access to the source domain data. In this paper, we propose and explore a new multi-modal extension of test-time adaptation for 3D semantic segmentation. We find that directly applying existing methods usually results in performance instability at test time because multi-modal input is not considered jointly. To design a framework that can take full advantage of multi-modality, where each modality provides regularized self-supervisory signals to other modalities, we propose two complementary modules within and across the modalities. First, Intra-modal Pseudolabel Generation (Intra-PG) is introduced to obtain reliable pseudo labels within each modality by aggregating information from two models that are both pre-trained on source data but updated with target data at different paces. Second, Inter-modal Pseudo-label Refinement (Inter-PR) adaptively selects more reliable pseudo labels from different modalities based on a proposed consistency scheme. Experiments demonstrate that our regularized pseudo labels produce stable self-learning signals in numerous multi-modal test-time adaptation scenarios for 3D semantic segmentation. Visit our project website at https://www.nec-labs.com/~mas/MM-TTA.