# Every Dataset Counts: Scaling up Monocular 3D Object Detection with Joint Datasets Training

arXiv: https://arxiv.org/abs/2310.00920 · HTML: https://mlsanity.com/paper/2310.00920/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 34 references that are in the map.
- First posted on arXiv 2 Oct 2023.

## Builds on (most cited first, 26 of 26)

- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Deep Layer Aggregation](https://mlsanity.com/paper/1707.06484/) (2017)
- [M3D-RPN: Monocular 3D Region Proposal Network for Object Detection](https://mlsanity.com/paper/1907.06038/) (2019)
- [Disentangling Monocular 3D Object Detection](https://mlsanity.com/paper/1905.12365/) (2019)
- [SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation](https://mlsanity.com/paper/2002.10111/) (2020)
- [Objects are Different: Flexible Monocular 3D Object Detection](https://mlsanity.com/paper/2104.02323/) (2021)
- [Learning Depth-Guided Convolutions for Monocular 3D Object Detection](https://mlsanity.com/paper/1912.04799/) (2019)
- [MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships](https://mlsanity.com/paper/2003.00504/) (2020)
- [RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving](https://mlsanity.com/paper/2001.03343/) (2020)
- [Accurate Monocular Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving](https://mlsanity.com/paper/1903.11444/) (2019)
- [Monocular 3D Object Detection with Pseudo-LiDAR Point Cloud](https://mlsanity.com/paper/1903.09847/) (2019)
- [Monocular 3D Object Detection Leveraging Accurate Proposals and Shape Reconstruction](https://mlsanity.com/paper/1904.01690/) (2019)
- [Autolabeling 3D Objects with Differentiable Rendering of SDF Shape Priors](https://mlsanity.com/paper/1911.11288/) (2019)
- [3D Object Detection from Images for Autonomous Driving: A Survey](https://mlsanity.com/paper/2202.02980/) (2022)
- [Weakly Supervised 3D Object Detection from Point Clouds](https://mlsanity.com/paper/2007.13970/) (2020)
- [Cityscapes 3D: Dataset and Benchmark for 9 DoF Vehicle Detection](https://mlsanity.com/paper/2006.07864/) (2020)
- [YOLOStereo3D: A Step Back to 2D for Efficient Stereo 3D Detection](https://mlsanity.com/paper/2103.09422/) (2021)
- [Monocular 3D Object Detection and Box Fitting Trained End-to-End Using Intersection-over-Union Loss](https://mlsanity.com/paper/1906.08070/) (2019)
- [Lidar Point Cloud Guided Monocular 3D Object Detection](https://mlsanity.com/paper/2104.09035/) (2021)
- [Shift R-CNN: Deep Monocular 3D Object Detection with Closed-Form Geometric Constraints](https://mlsanity.com/paper/1905.09970/) (2019)
- [RefinedMPL: Refined Monocular PseudoLiDAR for 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/1911.09712/) (2019)
- [WeakM3D: Towards Weakly Supervised Monocular 3D Object Detection](https://mlsanity.com/paper/2203.08332/) (2022)
- [Weakly Supervised Monocular 3D Object Detection using Multi-View Projection and Direction Consistency](https://mlsanity.com/paper/2303.08686/) (2023)
- [Focal Loss in 3D Object Detection](https://mlsanity.com/paper/1809.06065/) (2018)
- [Monocular 3D Detection with Geometric Constraints Embedding and Semi-supervised Training](https://mlsanity.com/paper/2009.00764/) (2020)

## Built on by (most cited first, 5 of 5)

- [Monocular 3D lane detection for Autonomous Driving: Recent Achievements, Challenges, and Outlooks](https://mlsanity.com/paper/2404.06860/) (2024)
- [CurbNet: Curb Detection Framework Based on LiDAR Point Cloud Segmentation](https://mlsanity.com/paper/2403.16794/) (2024)
- [TSCLIP: Robust CLIP Fine-Tuning for Worldwide Cross-Regional Traffic Sign Recognition](https://mlsanity.com/paper/2409.15077/) (2024)
- [CLRKDNet: Speeding up Lane Detection with Knowledge Distillation](https://mlsanity.com/paper/2405.12503/) (2024)
- [SFVO: Decoupled Confidence-Guided Stereo-Flow Visual Odometry with Bidirectional PnP](https://mlsanity.com/paper/2609.21754/) (2026)

## Abstract (arXiv)

Monocular 3D object detection plays a crucial role in autonomous driving. However, existing monocular 3D detection algorithms depend on 3D labels derived from LiDAR measurements, which are costly to acquire for new datasets and challenging to deploy in novel environments. Specifically, this study investigates the pipeline for training a monocular 3D object detection model on a diverse collection of 3D and 2D datasets. The proposed framework comprises three components: (1) a robust monocular 3D model capable of functioning across various camera settings, (2) a selective-training strategy to accommodate datasets with differing class annotations, and (3) a pseudo 3D training approach using 2D labels to enhance detection performance in scenes containing only 2D labels. With this framework, we could train models on a joint set of various open 3D/2D datasets to obtain models with significantly stronger generalization capability and enhanced performance on new dataset with only 2D labels. We conduct extensive experiments on KITTI/nuScenes/ONCE/Cityscapes/BDD100K datasets to demonstrate the scaling ability of the proposed method.