# End-to-end Learning for Joint Depth and Image Reconstruction from Diffracted Rotation

arXiv: https://arxiv.org/abs/2204.07076 · HTML: https://mlsanity.com/paper/2204.07076/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 57 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture](https://mlsanity.com/paper/1411.4734/) (2014)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [ResUNet-a: a deep learning framework for semantic segmentation of remotely sensed data](https://mlsanity.com/paper/1904.00592/) (2019)
- [Enforcing geometric constraints of virtual normal for depth prediction](https://mlsanity.com/paper/1907.12209/) (2019)
- [High Quality Monocular Depth Estimation via Transfer Learning](https://mlsanity.com/paper/1812.11941/) (2018)
- [Guiding Monocular Depth Estimation Using Depth-Attention Volume](https://mlsanity.com/paper/2004.02760/) (2020)
- [Deep Optics for Monocular Depth Estimation and 3D Object Detection](https://mlsanity.com/paper/1904.08601/) (2019)
- [Structure-Aware Residual Pyramid Network for Monocular Depth Estimation](https://mlsanity.com/paper/1907.06023/) (2019)
- [Detail Preserving Depth Estimation from a Single Image Using Attention Guided Networks](https://mlsanity.com/paper/1809.00646/) (2018)
- [Learning Deep Gradient Descent Optimization for Image Deconvolution](https://mlsanity.com/paper/1804.03368/) (2018)
- [Modeling Realistic Degradations in Non-blind Deconvolution](https://mlsanity.com/paper/1806.01097/) (2018)

## Built on by (most cited first, 4 of 4)

- [Can Language Understand Depth?](https://mlsanity.com/paper/2207.01077/) (2022)
- [WorDepth: Variational Language Prior for Monocular Depth Estimation](https://mlsanity.com/paper/2404.03635/) (2024)
- [Learning to Adapt CLIP for Few-Shot Monocular Depth Estimation](https://mlsanity.com/paper/2311.01034/) (2023)
- [Large-FOV RGBD Imaging via Structured PSF Coding in a Bio-inspired Monocentric System](https://mlsanity.com/paper/2510.25314/) (2025)

## Abstract (arXiv)

Monocular depth estimation is still an open challenge due to the ill-posed nature of the problem at hand. Deep learning based techniques have been extensively studied and proved capable of producing acceptable depth estimation accuracy even if the lack of meaningful and robust depth cues within single RGB input images severally limits their performance. Coded aperture-based methods using phase and amplitude masks encode strong depth cues within 2D images by means of depth-dependent Point Spread Functions (PSFs) at the price of a reduced image quality. In this paper, we propose a novel end-to-end learning approach for depth from diffracted rotation. A phase mask that produces a Rotating Point Spread Function (RPSF) as a function of defocus is jointly optimized with the weights of a depth estimation neural network. To this aim, we introduce a differentiable physical model of the aperture mask and exploit an accurate simulation of the camera imaging pipeline. Our approach requires a significantly less complex model and less training data, yet it is superior to existing methods in the task of monocular depth estimation on indoor benchmarks. In addition, we address the problem of image degradation by incorporating a non-blind and non-uniform image deblurring module to recover the sharp all-in-focus image from its RPSF-blurred counterpart.