# Scalable Vision-Based 3D Object Detection and Monocular Depth Estimation for Autonomous Driving

arXiv: https://arxiv.org/abs/2403.02037 · HTML: https://mlsanity.com/paper/2403.02037/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 116 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 28 of 28)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [DIODE: A Dense Indoor and Outdoor DEpth Dataset](https://mlsanity.com/paper/1908.00463/) (2019)
- [One Million Scenes for Autonomous Driving: ONCE Dataset](https://mlsanity.com/paper/2106.11037/) (2021)
- [Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/1906.06310/) (2019)
- [Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches](https://mlsanity.com/paper/1510.05970/) (2015)
- [How do neural networks see depth in single images?](https://mlsanity.com/paper/1905.07005/) (2019)
- [SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation](https://mlsanity.com/paper/2204.03636/) (2022)
- [MonoDistill: Learning Spatial Features for Monocular 3D Object Detection](https://mlsanity.com/paper/2201.10830/) (2022)
- [Adaptive Unimodal Cost Volume Filtering for Deep Stereo Matching](https://mlsanity.com/paper/1909.03751/) (2019)
- [Bidirectional Attention Network for Monocular Depth Estimation](https://mlsanity.com/paper/2009.00743/) (2020)
- [Ensemble Knowledge Distillation for Learning Improved and Efficient Networks](https://mlsanity.com/paper/1909.08097/) (2019)
- [Cityscapes 3D: Dataset and Benchmark for 9 DoF Vehicle Detection](https://mlsanity.com/paper/2006.07864/) (2020)
- [FADNet: A Fast and Accurate Network for Disparity Estimation](https://mlsanity.com/paper/2003.10758/) (2020)
- [Monocular 3D Object Detection and Box Fitting Trained End-to-End Using Intersection-over-Union Loss](https://mlsanity.com/paper/1906.08070/) (2019)
- [Object-Centric Stereo Matching for 3D Object Detection](https://mlsanity.com/paper/1909.07566/) (2019)
- [ZoomNet: Part-Aware Adaptive Zooming Neural Network for 3D Object Detection](https://mlsanity.com/paper/2003.00529/) (2020)
- [RefinedMPL: Refined Monocular PseudoLiDAR for 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/1911.09712/) (2019)
- [Knowledge distillation via adaptive instance normalization](https://mlsanity.com/paper/2003.04289/) (2020)

## Abstract (arXiv)

This dissertation is a multifaceted contribution to the advancement of vision-based 3D perception technologies. In the first segment, the thesis introduces structural enhancements to both monocular and stereo 3D object detection algorithms. By integrating ground-referenced geometric priors into monocular detection models, this research achieves unparalleled accuracy in benchmark evaluations for monocular 3D detection. Concurrently, the work refines stereo 3D detection paradigms by incorporating insights and inferential structures gleaned from monocular networks, thereby augmenting the operational efficiency of stereo detection systems. The second segment is devoted to data-driven strategies and their real-world applications in 3D vision detection. A novel training regimen is introduced that amalgamates datasets annotated with either 2D or 3D labels. This approach not only augments the detection models through the utilization of a substantially expanded dataset but also facilitates economical model deployment in real-world scenarios where only 2D annotations are readily available. Lastly, the dissertation presents an innovative pipeline tailored for unsupervised depth estimation in autonomous driving contexts. Extensive empirical analyses affirm the robustness and efficacy of this newly proposed pipeline. Collectively, these contributions lay a robust foundation for the widespread adoption of vision-based 3D perception technologies in autonomous driving applications.