# Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection

arXiv: https://arxiv.org/abs/2303.06880 · HTML: https://mlsanity.com/paper/2303.06880/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 40 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Builds on (most cited first, 30 of 35)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [Frustum PointNets for 3D Object Detection from RGB-D Data](https://mlsanity.com/paper/1711.08488/) (2017)
- [Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection](https://mlsanity.com/paper/2012.15712/) (2020)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [PIXOR: Real-time 3D Object Detection from Point Clouds](https://mlsanity.com/paper/1902.06326/) (2019)
- [STD: Sparse-to-Dense 3D Object Detector for Point Cloud](https://mlsanity.com/paper/1907.10471/) (2019)
- [Voxel Transformer for 3D Object Detection](https://mlsanity.com/paper/2109.02497/) (2021)
- [PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection](https://mlsanity.com/paper/2102.00463/) (2021)
- [Dynamic Head: Unifying Object Detection Heads with Attentions](https://mlsanity.com/paper/2106.08322/) (2021)
- [MVX-Net: Multimodal VoxelNet for 3D Object Detection](https://mlsanity.com/paper/1904.01649/) (2019)
- [ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection](https://mlsanity.com/paper/2103.05346/) (2021)
- [Not All Points Are Equal: Learning Highly Efficient Point-based Detectors for 3D LiDAR Point Clouds](https://mlsanity.com/paper/2203.11139/) (2022)
- [Simple multi-dataset detection](https://mlsanity.com/paper/2102.13086/) (2021)
- [MSeg: A Composite Dataset for Multi-domain Semantic Segmentation](https://mlsanity.com/paper/2112.13762/) (2021)
- [SPG: Unsupervised Domain Adaptation for 3D Object Detection via Semantic Point Generation](https://mlsanity.com/paper/2108.06709/) (2021)
- [Towards Universal Object Detection by Domain Attention](https://mlsanity.com/paper/1904.04402/) (2019)

## Built on by (most cited first, 30 of 39)

- [EMR-Merging: Tuning-Free High-Performance Model Merging](https://mlsanity.com/paper/2405.17461/) (2024)
- [Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training](https://mlsanity.com/paper/2308.09718/) (2023)
- [LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences](https://mlsanity.com/paper/2412.01292/) (2024)
- [Uni3DETR: Unified 3D Detection Transformer](https://mlsanity.com/paper/2310.05699/) (2023)
- [Multi-Space Alignments Towards Universal LiDAR Segmentation](https://mlsanity.com/paper/2405.01538/) (2024)
- [GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training](https://mlsanity.com/paper/2412.11863/) (2024)
- [AD-PT: Autonomous Driving Pre-Training with Large-scale Point Cloud Dataset](https://mlsanity.com/paper/2306.00612/) (2023)
- [Perspective-Invariant 3D Object Detection](https://mlsanity.com/paper/2507.17665/) (2025)
- [Merging Vision Transformers from Different Tasks and Domains](https://mlsanity.com/paper/2312.16240/) (2023)
- [MS3D++: Ensemble of Experts for Multi-Source Unsupervised Domain Adaptation in 3D Object Detection](https://mlsanity.com/paper/2308.05988/) (2023)
- [ReSimAD: Zero-Shot 3D Domain Transfer for Autonomous Driving with Source Reconstruction and Target Simulation](https://mlsanity.com/paper/2309.05527/) (2023)
- [MS3D: Leveraging Multiple Detectors for Unsupervised Domain Adaptation in 3D Object Detection](https://mlsanity.com/paper/2304.02431/) (2023)
- [Multimodal 3D Object Detection on Unseen Domains](https://mlsanity.com/paper/2404.11764/) (2024)
- [One for All: Multi-Domain Joint Training for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/2411.01584/) (2024)
- [Scaling Multi-Camera 3D Object Detection through Weak-to-Strong Eliciting](https://mlsanity.com/paper/2404.06700/) (2024)
- [Domain Generalization of 3D Object Detection by Density-Resampling](https://mlsanity.com/paper/2311.10845/) (2023)
- [Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning](https://mlsanity.com/paper/2508.01651/) (2025)
- [SPOT: Scalable 3D Pre-training via Occupancy Prediction for Learning Transferable 3D Representations](https://mlsanity.com/paper/2309.10527/) (2023)
- [Exploiting GPT-4 Vision for Zero-shot Point Cloud Understanding](https://mlsanity.com/paper/2401.07572/) (2024)
- [UniGS: Unified Language-Image-3D Pretraining with Gaussian Splatting](https://mlsanity.com/paper/2502.17860/) (2025)
- [Detect Closer Surfaces that can be Seen: New Modeling and Evaluation in Cross-domain 3D Object Detection](https://mlsanity.com/paper/2407.04061/) (2024)
- [Street Gaussians without 3D Object Tracker](https://mlsanity.com/paper/2412.05548/) (2024)
- [Uni2Det: Unified and Universal Framework for Prompt-Guided Multi-dataset 3D Detection](https://mlsanity.com/paper/2409.20558/) (2024)
- [Foundation Model-Powered 3D Few-Shot Class Incremental Learning via Training-free Adaptor](https://mlsanity.com/paper/2410.09237/) (2024)
- [Synth It Like KITTI: Synthetic Data Generation for Object Detection in Driving Scenarios](https://mlsanity.com/paper/2502.15076/) (2025)
- [Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Data](https://mlsanity.com/paper/2506.09093/) (2025)
- [MergeOcc: Bridge the Domain Gap between Different LiDARs for Robust Occupancy Prediction](https://mlsanity.com/paper/2403.08512/) (2024)
- [VideoAfford: Grounding 3D Affordance from Human-Object-Interaction Videos via Multimodal Large Language Model](https://mlsanity.com/paper/2602.09638/) (2026)
- [OmniLiDAR: A Unified Diffusion Framework for Multi-Domain 3D LiDAR Generation](https://mlsanity.com/paper/2605.13815/) (2026)
- [Multi-view Vision-Prompt Fusion Network: Can 2D Pre-trained Model Boost 3D Point Cloud Data-scarce Learning?](https://mlsanity.com/paper/2304.10224/) (2023)

## Abstract (arXiv)

Current 3D object detection models follow a single dataset-specific training and testing paradigm, which often faces a serious detection accuracy drop when they are directly deployed in another dataset. In this paper, we study the task of training a unified 3D detector from multiple datasets. We observe that this appears to be a challenging task, which is mainly due to that these datasets present substantial data-level differences and taxonomy-level variations caused by different LiDAR types and data acquisition standards. Inspired by such observation, we present a Uni3D which leverages a simple data-level correction operation and a designed semantic-level coupling-and-recoupling module to alleviate the unavoidable data-level and taxonomy-level differences, respectively. Our method is simple and easily combined with many 3D object detection baselines such as PV-RCNN and Voxel-RCNN, enabling them to effectively learn from multiple off-the-shelf 3D datasets to obtain more discriminative and generalizable representations. Experiments are conducted on many dataset consolidation settings including Waymo-nuScenes, nuScenes-KITTI, Waymo-KITTI, and Waymo-nuScenes-KITTI consolidations. Their results demonstrate that Uni3D exceeds a series of individual detectors trained on a single dataset, with a 1.04x parameter increase over a selected baseline detector. We expect this work will inspire the research of 3D generalization since it will push the limits of perceptual performance.