# Deep Optics for Monocular Depth Estimation and 3D Object Detection

arXiv: https://arxiv.org/abs/1904.08601 · HTML: https://mlsanity.com/paper/1904.08601/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Apr 2019.

## Built on by (most cited first, 30 of 65)

- [3D Object Detection for Autonomous Driving: A Comprehensive Survey](https://mlsanity.com/paper/2206.09474/) (2022)
- [3D Object Detection from Images for Autonomous Driving: A Survey](https://mlsanity.com/paper/2202.02980/) (2022)
- [Curriculum Learning for ab initio Deep Learned Refractive Optics](https://mlsanity.com/paper/2302.01089/) (2023)
- [Time-Multiplexed Coded Aperture Imaging: Learned Coded Aperture and Pixel Exposures for Compressive Imaging Systems](https://mlsanity.com/paper/2104.02820/) (2021)
- [The Differentiable Lens: Compound Lens Search over Glass Surfaces and Materials for Object Detection](https://mlsanity.com/paper/2212.04441/) (2022)
- [Thin On-Sensor Nanophotonic Array Cameras](https://mlsanity.com/paper/2308.02797/) (2023)
- [Learning Spatially Varying Pixel Exposures for Motion Deblurring](https://mlsanity.com/paper/2204.07267/) (2022)
- [Aberration-Aware Depth-from-Focus](https://mlsanity.com/paper/2303.04654/) (2023)
- [End-to-End Hybrid Refractive-Diffractive Lens Design with Differentiable Ray-Wave Model](https://mlsanity.com/paper/2406.00834/) (2024)
- [CodedStereo: Learned Phase Masks for Large Depth-of-field Stereo](https://mlsanity.com/paper/2104.04641/) (2021)
- [Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging](https://mlsanity.com/paper/2103.16693/) (2021)
- [Task-Driven Lens Design](https://mlsanity.com/paper/2305.17185/) (2023)
- [Deep Optics for Video Snapshot Compressive Imaging](https://mlsanity.com/paper/2404.05274/) (2024)
- [X-view: Non-egocentric Multi-View 3D Object Detector](https://mlsanity.com/paper/2103.13001/) (2021)
- [Computational Imaging and Artificial Intelligence: The Next Revolution of Mobile Vision](https://mlsanity.com/paper/2109.08880/) (2021)
- [Depth Estimation and Image Restoration by Deep Learning from Defocused Images](https://mlsanity.com/paper/2302.10730/) (2023)
- [Exploring Quasi-Global Solutions to Compound Lens Based Computational Imaging Systems](https://mlsanity.com/paper/2404.19201/) (2024)
- [CodedEvents: Optimal Point-Spread-Function Engineering for 3D-Tracking with Event Cameras](https://mlsanity.com/paper/2406.09409/) (2024)
- [Physics vs. Learned Priors: Rethinking Camera and Algorithm Design for Task-Specific Imaging](https://mlsanity.com/paper/2204.09871/) (2022)
- [3D Scene Inference from Transient Histograms](https://mlsanity.com/paper/2211.05094/) (2022)
- [Differentiable Sensor Layouts for End-to-End Learning of Task-Specific Camera Parameters](https://mlsanity.com/paper/2304.14736/) (2023)
- [DISeR: Designing Imaging Systems with Reinforcement Learning](https://mlsanity.com/paper/2309.13851/) (2023)
- [Video Reconstruction from a Single Motion Blurred Image using Learned Dynamic Phase Coding](https://mlsanity.com/paper/2112.14768/) (2021)
- [TiDy-PSFs: Computational Imaging with Time-Averaged Dynamic Point-Spread-Functions](https://mlsanity.com/paper/2303.17583/) (2023)
- [Minimalist Vision with Freeform Pixels](https://mlsanity.com/paper/2501.00142/) (2024)
- [A Simple Framework for 3D Lensless Imaging with Programmable Masks](https://mlsanity.com/paper/2108.07966/) (2021)
- [End-to-end Learning for Joint Depth and Image Reconstruction from Diffracted Rotation](https://mlsanity.com/paper/2204.07076/) (2022)
- [PS2F: Polarized Spiral Point Spread Function for Single-Shot 3D Sensing](https://mlsanity.com/paper/2207.00945/) (2022)
- [Blurry-Edges: Photon-Limited Depth Estimation from Defocused Boundaries](https://mlsanity.com/paper/2503.23606/) (2025)
- [FourierNets enable the design of highly non-local optical encoders for computational imaging](https://mlsanity.com/paper/2104.10611/) (2021)

## Abstract (arXiv)

Depth estimation and 3D object detection are critical for scene understanding but remain challenging to perform with a single image due to the loss of 3D information during image capture. Recent models using deep neural networks have improved monocular depth estimation performance, but there is still difficulty in predicting absolute depth and generalizing outside a standard dataset. Here we introduce the paradigm of deep optics, i.e. end-to-end design of optics and image processing, to the monocular depth estimation problem, using coded defocus blur as an additional depth cue to be decoded by a neural network. We evaluate several optical coding strategies along with an end-to-end optimization scheme for depth estimation on three datasets, including NYU Depth v2 and KITTI. We find an optimized freeform lens design yields the best results, but chromatic aberration from a singlet lens offers significantly improved performance as well. We build a physical prototype and validate that chromatic aberrations improve depth estimation on real-world results. In addition, we train object detection networks on the KITTI dataset and show that the lens optimized for depth estimation also results in improved 3D object detection performance.