# MDT3D: Multi-Dataset Training for LiDAR 3D Object Detection Generalization

arXiv: https://arxiv.org/abs/2308.01000 · HTML: https://mlsanity.com/paper/2308.01000/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 26 references that are in the map.
- First posted on arXiv 2 Aug 2023.

## Builds on (most cited first, 22 of 22)

- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay](https://mlsanity.com/paper/1803.09820/) (2018)
- [One Million Scenes for Autonomous Driving: ONCE Dataset](https://mlsanity.com/paper/2106.11037/) (2021)
- [ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection](https://mlsanity.com/paper/2103.05346/) (2021)
- [Train in Germany, Test in The USA: Making 3D Object Detectors Generalize](https://mlsanity.com/paper/2005.08139/) (2020)
- [Simple multi-dataset detection](https://mlsanity.com/paper/2102.13086/) (2021)
- [SPG: Unsupervised Domain Adaptation for 3D Object Detection via Semantic Point Generation](https://mlsanity.com/paper/2108.06709/) (2021)
- [Object Detection with a Unified Label Space from Multiple Datasets](https://mlsanity.com/paper/2008.06614/) (2020)
- [3D-VField: Adversarial Augmentation of Point Clouds for Domain Generalization in 3D Object Detection](https://mlsanity.com/paper/2112.04764/) (2021)
- [Towards Generalization Across Depth for Monocular 3D Object Detection](https://mlsanity.com/paper/1912.08035/) (2019)
- [Towards Domain Generalization in Object Detection](https://mlsanity.com/paper/2203.14387/) (2022)
- [COLA: COarse LAbel pre-training for 3D semantic segmentation of sparse LiDAR datasets](https://mlsanity.com/paper/2202.06884/) (2022)
- [Understanding the Domain Gap in LiDAR Object Detection Networks](https://mlsanity.com/paper/2204.10024/) (2022)

## Built on by (most cited first, 14 of 14)

- [Multi-Space Alignments Towards Universal LiDAR Segmentation](https://mlsanity.com/paper/2405.01538/) (2024)
- [UniDet3D: Multi-dataset Indoor 3D Object Detection](https://mlsanity.com/paper/2409.04234/) (2024)
- [LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving](https://mlsanity.com/paper/2501.04005/) (2025)
- [COLA: COarse-LAbel multi-source LiDAR semantic segmentation for autonomous driving](https://mlsanity.com/paper/2311.03017/) (2023)
- [Domain Generalization of 3D Object Detection by Density-Resampling](https://mlsanity.com/paper/2311.10845/) (2023)
- [ParisLuco3D: A high-quality target dataset for domain generalization of LiDAR perception](https://mlsanity.com/paper/2310.16542/) (2023)
- [Street Gaussians without 3D Object Tracker](https://mlsanity.com/paper/2412.05548/) (2024)
- [GBlobs: Explicit Local Structure via Gaussian Blobs for Improved Cross-Domain LiDAR-based 3D Object Detection](https://mlsanity.com/paper/2503.08639/) (2025)
- [OmniLiDAR: A Unified Diffusion Framework for Multi-Domain 3D LiDAR Generation](https://mlsanity.com/paper/2605.13815/) (2026)
- [AMA-SAM: Adversarial Multi-Domain Alignment of Segment Anything Model for High-Fidelity Histology Nuclei Segmentation](https://mlsanity.com/paper/2503.21695/) (2025)
- [Domain Adaptation for Different Sensor Configurations in 3D Object Detection](https://mlsanity.com/paper/2509.04711/) (2025)
- [DoReMi: Bridging 3D Domains via Topology-Aware Domain-Representation Mixture of Experts](https://mlsanity.com/paper/2511.11232/) (2025)
- [UniFlow: Zero-Shot LiDAR Scene Flow for Autonomous Vehicles](https://mlsanity.com/paper/2511.18254/) (2025)
- [STAR: A Spatial-Topology Aware Routing Framework for Generalizable 3D Scene Understanding](https://mlsanity.com/paper/2608.11699/) (2026)

## Abstract (arXiv)

Supervised 3D Object Detection models have been displaying increasingly better performance in single-domain cases where the training data comes from the same environment and sensor as the testing data. However, in real-world scenarios data from the target domain may not be available for finetuning or for domain adaptation methods. Indeed, 3D object detection models trained on a source dataset with a specific point distribution have shown difficulties in generalizing to unseen datasets. Therefore, we decided to leverage the information available from several annotated source datasets with our Multi-Dataset Training for 3D Object Detection (MDT3D) method to increase the robustness of 3D object detection models when tested in a new environment with a different sensor configuration. To tackle the labelling gap between datasets, we used a new label mapping based on coarse labels. Furthermore, we show how we managed the mix of datasets during training and finally introduce a new cross-dataset augmentation method: cross-dataset object injection. We demonstrate that this training paradigm shows improvements for different types of 3D object detection models. The source code and additional results for this research project will be publicly available on GitHub for interested parties to access and utilize: https://github.com/LouisSF/MDT3D