# Depth Is All You Need for Monocular 3D Detection

arXiv: https://arxiv.org/abs/2210.02493 · HTML: https://mlsanity.com/paper/2210.02493/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 58 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Builds on (most cited first, 30 of 50)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [PointPillars: Fast Encoders for Object Detection from Point Clouds](https://mlsanity.com/paper/1812.05784/) (2018)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries](https://mlsanity.com/paper/2110.06922/) (2021)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving](https://mlsanity.com/paper/1812.07179/) (2018)
- [FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection](https://mlsanity.com/paper/2104.10956/) (2021)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Virtual Worlds as Proxy for Multi-Object Tracking Analysis](https://mlsanity.com/paper/1605.06457/) (2016)
- [M3D-RPN: Monocular 3D Region Proposal Network for Object Detection](https://mlsanity.com/paper/1907.06038/) (2019)
- [Is Pseudo-Lidar needed for Monocular 3D Object detection?](https://mlsanity.com/paper/2108.06417/) (2021)
- [Disentangling Monocular 3D Object Detection](https://mlsanity.com/paper/1905.12365/) (2019)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation](https://mlsanity.com/paper/2002.10111/) (2020)
- [CenterMask : Real-Time Anchor-Free Instance Segmentation](https://mlsanity.com/paper/1911.06667/) (2019)
- [Probabilistic and Geometric Depth: Detecting Objects in Perspective](https://mlsanity.com/paper/2107.14160/) (2021)
- [Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving](https://mlsanity.com/paper/1906.06310/) (2019)
- [Objects are Different: Flexible Monocular 3D Object Detection](https://mlsanity.com/paper/2104.02323/) (2021)
- [Learning Depth-Guided Convolutions for Monocular 3D Object Detection](https://mlsanity.com/paper/1912.04799/) (2019)

## Built on by (most cited first, 4 of 4)

- [MonoTAKD: Teaching Assistant Knowledge Distillation for Monocular 3D Object Detection](https://mlsanity.com/paper/2404.04910/) (2024)
- [FlatFusion: Delving into Details of Sparse Transformer-based Camera-LiDAR Fusion for Autonomous Driving](https://mlsanity.com/paper/2408.06832/) (2024)
- [Multi-View Attentive Contextualization for Multi-View 3D Object Detection](https://mlsanity.com/paper/2405.12200/) (2024)
- [MonoCT: Overcoming Monocular 3D Detection Domain Shift with Consistent Teacher Models](https://mlsanity.com/paper/2503.13743/) (2025)

## Abstract (arXiv)

A key contributor to recent progress in 3D detection from single images is monocular depth estimation. Existing methods focus on how to leverage depth explicitly, by generating pseudo-pointclouds or providing attention cues for image features. More recent works leverage depth prediction as a pretraining task and fine-tune the depth representation while training it for 3D detection. However, the adaptation is insufficient and is limited in scale by manual labels. In this work, we propose to further align depth representation with the target domain in unsupervised fashions. Our methods leverage commonly available LiDAR or RGB videos during training time to fine-tune the depth representation, which leads to improved 3D detectors. Especially when using RGB videos, we show that our two-stage training by first generating pseudo-depth labels is critical because of the inconsistency in loss distribution between the two tasks. With either type of reference data, our multi-task learning approach improves over the state of the art on both KITTI and NuScenes, while matching the test-time complexity of its single task sub-network.