# Uni2Det: Unified and Universal Framework for Prompt-Guided Multi-dataset 3D Detection

arXiv: https://arxiv.org/abs/2409.20558 · HTML: https://mlsanity.com/paper/2409.20558/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 33 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Builds on (most cited first, 29 of 29)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection](https://mlsanity.com/paper/1711.06396/) (2017)
- [Center-based 3D Object Detection and Tracking](https://mlsanity.com/paper/2006.11275/) (2020)
- [PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud](https://mlsanity.com/paper/1812.04244/) (2018)
- [PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection](https://mlsanity.com/paper/1912.13192/) (2019)
- [Multi-View 3D Object Detection Network for Autonomous Driving](https://mlsanity.com/paper/1611.07759/) (2016)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [3DSSD: Point-based 3D Single Stage Object Detector](https://mlsanity.com/paper/2002.10187/) (2020)
- [Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection](https://mlsanity.com/paper/2012.15712/) (2020)
- [From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network](https://mlsanity.com/paper/1907.03670/) (2019)
- [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](https://mlsanity.com/paper/2212.02499/) (2022)
- [PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection](https://mlsanity.com/paper/2102.00463/) (2021)
- [Self-supervised Pretraining of Visual Features in the Wild](https://mlsanity.com/paper/2103.01988/) (2021)
- [Dynamic Head: Unifying Object Detection Heads with Attentions](https://mlsanity.com/paper/2106.08322/) (2021)
- [ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection](https://mlsanity.com/paper/2103.05346/) (2021)
- [Train in Germany, Test in The USA: Making 3D Object Detectors Generalize](https://mlsanity.com/paper/2005.08139/) (2020)
- [Simple multi-dataset detection](https://mlsanity.com/paper/2102.13086/) (2021)
- [Task Residual for Tuning Vision-Language Models](https://mlsanity.com/paper/2211.10277/) (2022)
- [MSeg: A Composite Dataset for Multi-domain Semantic Segmentation](https://mlsanity.com/paper/2112.13762/) (2021)
- [Towards Universal Object Detection by Domain Attention](https://mlsanity.com/paper/1904.04402/) (2019)
- [Towards Large-scale 3D Representation Learning with Multi-dataset Point Prompt Training](https://mlsanity.com/paper/2308.09718/) (2023)
- [Object Detection with a Unified Label Space from Multiple Datasets](https://mlsanity.com/paper/2008.06614/) (2020)
- [ST3D++: Denoised Self-training for Unsupervised Domain Adaptation on 3D Object Detection](https://mlsanity.com/paper/2108.06682/) (2021)
- [LiDAR Distillation: Bridging the Beam-Induced Domain Gap for 3D Object Detection](https://mlsanity.com/paper/2203.14956/) (2022)
- [Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection](https://mlsanity.com/paper/2303.06880/) (2023)
- [mDALU: Multi-Source Domain Adaptation and Label Unification with Partial Datasets](https://mlsanity.com/paper/2012.08385/) (2020)
- [Universal-RCNN: Universal Object Detector via Transferable Graph R-CNN](https://mlsanity.com/paper/2002.07417/) (2020)

## Built on by (most cited first, 3 of 3)

- [Hierarchical Cross-modal Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2507.14976/) (2025)
- [Adversarial Domain Prompt Tuning and Generation for Single Domain Generalization](https://mlsanity.com/paper/2606.21736/) (2026)
- [COME: Dual Structure-Semantic Learning with Collaborative MoE for Universal Lesion Detection Across Heterogeneous Ultrasound Datasets](https://mlsanity.com/paper/2508.09886/) (2025)

## Abstract (arXiv)

We present Uni2Det, a brand new framework for unified and universal multi-dataset training on 3D detection, enabling robust performance across diverse domains and generalization to unseen domains. Due to substantial disparities in data distribution and variations in taxonomy across diverse domains, training such a detector by simply merging datasets poses a significant challenge. Motivated by this observation, we introduce multi-stage prompting modules for multi-dataset 3D detection, which leverages prompts based on the characteristics of corresponding datasets to mitigate existing differences. This elegant design facilitates seamless plug-and-play integration within various advanced 3D detection frameworks in a unified manner, while also allowing straightforward adaptation for universal applicability across datasets. Experiments are conducted across multiple dataset consolidation scenarios involving KITTI, Waymo, and nuScenes, demonstrating that our Uni2Det outperforms existing methods by a large margin in multi-dataset training. Notably, results on zero-shot cross-dataset transfer validate the generalization capability of our proposed method.