# CTS: Sim-to-Real Unsupervised Domain Adaptation on 3D Detection

arXiv: https://arxiv.org/abs/2406.18129 · HTML: https://mlsanity.com/paper/2406.18129/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 26 Jun 2024.

## Builds on (most cited first, 24 of 24)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Learning Transferable Features with Deep Adaptation Networks](https://mlsanity.com/paper/1502.02791/) (2015)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [OPV2V: An Open Benchmark Dataset and Fusion Pipeline for Perception with Vehicle-to-Vehicle Communication](https://mlsanity.com/paper/2109.07644/) (2021)
- [Unbiased Mean Teacher for Cross-domain Object Detection](https://mlsanity.com/paper/2003.00707/) (2020)
- [ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection](https://mlsanity.com/paper/2103.05346/) (2021)
- [Train in Germany, Test in The USA: Making 3D Object Detectors Generalize](https://mlsanity.com/paper/2005.08139/) (2020)
- [SHIFT: A Synthetic Driving Dataset for Continuous Multi-Task Domain Adaptation](https://mlsanity.com/paper/2206.08367/) (2022)
- [LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving](https://mlsanity.com/paper/1903.08701/) (2019)
- [Adversarial-Learned Loss for Domain Adaptation](https://mlsanity.com/paper/2001.01046/) (2020)
- [Unsupervised Domain Adaptive 3D Detection with Multi-Level Consistency](https://mlsanity.com/paper/2107.11355/) (2021)
- [ST3D++: Denoised Self-training for Unsupervised Domain Adaptation on 3D Object Detection](https://mlsanity.com/paper/2108.06682/) (2021)
- [Learning an Uncertainty-Aware Object Detector for Autonomous Driving](https://mlsanity.com/paper/1910.11375/) (2019)
- [Leveraging Heteroscedastic Aleatoric Uncertainties for Robust Real-Time LiDAR 3D Object Detection](https://mlsanity.com/paper/1809.05590/) (2018)
- [Uncertainty-aware Mean Teacher for Source-free Unsupervised Domain Adaptive 3D Object Detection](https://mlsanity.com/paper/2109.14651/) (2021)
- [Capture Uncertainties in Deep Neural Networks for Safe Operation of Autonomous Driving Vehicles](https://mlsanity.com/paper/2108.05118/) (2021)

## Built on by (most cited first, 2 of 2)

- [Towards Generating Realistic 3D Semantic Training Data for Autonomous Driving](https://mlsanity.com/paper/2503.21449/) (2025)
- [Unsupervised Adaptation from FDG to PSMA PET/CT for 3D Lesion Detection under Label Shift](https://mlsanity.com/paper/2603.13666/) (2026)

## Abstract (arXiv)

Simulation data can be accurately labeled and have been expected to improve the performance of data-driven algorithms, including object detection. However, due to the various domain inconsistencies from simulation to reality (sim-to-real),cross-domain object detection algorithms usually suffer from dramatic performance drops. While numerous unsupervised domain adaptation (UDA) methods have been developed to address cross-domain tasks between real-world datasets, progress in sim-to-real remains limited. This paper presents a novel Complex-to-Simple (CTS) framework to transfer models from labeled simulation (source) to unlabeled reality (target) domains. Based on a two-stage detector, the novelty of this work is threefold: 1) developing fixed-size anchor heads and RoI augmentation to address size bias and feature diversity between two domains, thereby improving the quality of pseudo-label; 2) developing a novel corner-format representation of aleatoric uncertainty (AU) for the bounding box, to uniformly quantify pseudo-label quality; 3) developing a noise-aware mean teacher domain adaptation method based on AU, as well as object-level and frame-level sampling strategies, to migrate the impact of noisy labels. Experimental results demonstrate that our proposed approach significantly enhances the sim-to-real domain adaptation capability of 3D object detection models, outperforming state-of-the-art cross-domain algorithms, which are usually developed for real-to-real UDA tasks.