# Passive Snapshot Coded Aperture Dual-Pixel RGB-D Imaging

arXiv: https://arxiv.org/abs/2402.18102 · HTML: https://mlsanity.com/paper/2402.18102/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 42 references that are in the map.
- First posted on arXiv 28 Feb 2024.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Road Extraction by Deep Residual U-Net](https://mlsanity.com/paper/1711.10684/) (2017)
- [Defocus Deblurring Using Dual-Pixel Data](https://mlsanity.com/paper/2005.00305/) (2020)
- [Synthetic Depth-of-Field with a Single-Camera Mobile Phone](https://mlsanity.com/paper/1806.04171/) (2018)
- [Learning Single Camera Depth Estimation using Dual-Pixels](https://mlsanity.com/paper/1904.05822/) (2019)
- [Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data](https://mlsanity.com/paper/2012.03255/) (2020)
- [Dual Pixel Exploration: Simultaneous Depth Estimation and Image Restoration](https://mlsanity.com/paper/2012.00301/) (2020)
- [Defocus Map Estimation and Deblurring from a Single Dual-Pixel Image](https://mlsanity.com/paper/2110.05655/) (2021)
- [Du2Net: Learning Depth Estimation from Dual-Cameras and Dual-Pixels](https://mlsanity.com/paper/2003.14299/) (2020)
- [Facial Depth and Normal Estimation using Single Dual-Pixel Camera](https://mlsanity.com/paper/2111.12928/) (2021)
- [Data Driven Coded Aperture Design for Depth Recovery](https://mlsanity.com/paper/1705.10021/) (2017)

## Built on by (most cited first, 6 of 6)

- [Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues](https://mlsanity.com/paper/2505.17358/) (2025)
- [Simulating Dual-Pixel Images From Ray Tracing For Depth Estimation](https://mlsanity.com/paper/2503.11213/) (2025)
- [Revisiting Disparity from Dual-Pixel Images: Physics-Informed Lightweight Depth Estimation](https://mlsanity.com/paper/2411.04714/) (2024)
- [Physically Grounded Monocular Depth via Nanophotonic Wavefront Encoding](https://mlsanity.com/paper/2503.15770/) (2025)
- [FoundDP: Revisiting Weak Disparity Observability in Dual-Pixel Depth Estimation](https://mlsanity.com/paper/2607.01900/) (2026)
- [Aperture-aware Dispersion 5-D Light-field Imaging Spectrometer](https://mlsanity.com/paper/2607.04635/) (2026)

## Abstract (arXiv)

Passive, compact, single-shot 3D sensing is useful in many application areas such as microscopy, medical imaging, surgical navigation, and autonomous driving where form factor, time, and power constraints can exist. Obtaining RGB-D scene information over a short imaging distance, in an ultra-compact form factor, and in a passive, snapshot manner is challenging. Dual-pixel (DP) sensors are a potential solution to achieve the same. DP sensors collect light rays from two different halves of the lens in two interleaved pixel arrays, thus capturing two slightly different views of the scene, like a stereo camera system. However, imaging with a DP sensor implies that the defocus blur size is directly proportional to the disparity seen between the views. This creates a trade-off between disparity estimation vs. deblurring accuracy. To improve this trade-off effect, we propose CADS (Coded Aperture Dual-Pixel Sensing), in which we use a coded aperture in the imaging lens along with a DP sensor. In our approach, we jointly learn an optimal coded pattern and the reconstruction algorithm in an end-to-end optimization setting. Our resulting CADS imaging system demonstrates improvement of >1.5dB PSNR in all-in-focus (AIF) estimates and 5-6% in depth estimation quality over naive DP sensing for a wide range of aperture settings. Furthermore, we build the proposed CADS prototypes for DSLR photography settings and in an endoscope and a dermoscope form factor. Our novel coded dual-pixel sensing approach demonstrates accurate RGB-D reconstruction results in simulations and real-world experiments in a passive, snapshot, and compact manner.