# Learning 2D to 3D Lifting for Object Detection in 3D for Autonomous Vehicles

arXiv: https://arxiv.org/abs/1904.08494 · HTML: https://mlsanity.com/paper/1904.08494/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Mar 2019.

## Built on by (most cited first, 18 of 18)

- [Categorical Depth Distribution Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2103.01100/) (2021)
- [Is Pseudo-Lidar needed for Monocular 3D Object detection?](https://mlsanity.com/paper/2108.06417/) (2021)
- [Vision-Centric BEV Perception: A Survey](https://mlsanity.com/paper/2208.02797/) (2022)
- [3D Object Detection from Images for Autonomous Driving: A Survey](https://mlsanity.com/paper/2202.02980/) (2022)
- [MonoNeRD: NeRF-like Representations for Monocular 3D Object Detection](https://mlsanity.com/paper/2308.09421/) (2023)
- [CramNet: Camera-Radar Fusion with Ray-Constrained Cross-Attention for Robust 3D Object Detection](https://mlsanity.com/paper/2210.09267/) (2022)
- [FGR: Frustum-Aware Geometric Reasoning for Weakly Supervised 3D Vehicle Detection](https://mlsanity.com/paper/2105.07647/) (2021)
- [MatrixVT: Efficient Multi-Camera to BEV Transformation for 3D Perception](https://mlsanity.com/paper/2211.10593/) (2022)
- [Beyond Image to Depth: Improving Depth Prediction using Echoes](https://mlsanity.com/paper/2103.08468/) (2021)
- [DPFT: Dual Perspective Fusion Transformer for Camera-Radar-based Object Detection](https://mlsanity.com/paper/2404.03015/) (2024)
- [MonoCInIS: Camera Independent Monocular 3D Object Detection using Instance Segmentation](https://mlsanity.com/paper/2110.00464/) (2021)
- [Monocular BEV Perception of Road Scenes via Front-to-Top View Projection](https://mlsanity.com/paper/2211.08144/) (2022)
- [Survey and Systematization of 3D Object Detection Models and Methods](https://mlsanity.com/paper/2201.09354/) (2022)
- [SBEVNet: End-to-End Deep Stereo Layout Estimation](https://mlsanity.com/paper/2105.11705/) (2021)
- [Scalable Primitives for Generalized Sensor Fusion in Autonomous Vehicles](https://mlsanity.com/paper/2112.00219/) (2021)
- [A Dual-Cycled Cross-View Transformer Network for Unified Road Layout Estimation and 3D Object Detection in the Bird's-Eye-View](https://mlsanity.com/paper/2209.08844/) (2022)
- [LAPTNet: LiDAR-Aided Perspective Transform Network](https://mlsanity.com/paper/2211.14445/) (2022)
- [BEVal: A Cross-dataset Evaluation Study of BEV Segmentation Models for Autonomous Driving](https://mlsanity.com/paper/2408.16322/) (2024)

## Abstract (arXiv)

We address the problem of 3D object detection from 2D monocular images in autonomous driving scenarios. We propose to lift the 2D images to 3D representations using learned neural networks and leverage existing networks working directly on 3D data to perform 3D object detection and localization. We show that, with carefully designed training mechanism and automatically selected minimally noisy data, such a method is not only feasible, but gives higher results than many methods working on actual 3D inputs acquired from physical sensors. On the challenging KITTI benchmark, we show that our 2D to 3D lifted method outperforms many recent competitive 3D networks while significantly outperforming previous state-of-the-art for 3D detection from monocular images. We also show that a late fusion of the output of the network trained on generated 3D images, with that trained on real 3D images, improves performance. We find the results very interesting and argue that such a method could serve as a highly reliable backup in case of malfunction of expensive 3D sensors, if not potentially making them redundant, at least in the case of low human injury risk autonomous navigation scenarios like warehouse automation.