# CodedStereo: Learned Phase Masks for Large Depth-of-field Stereo

arXiv: https://arxiv.org/abs/2104.04641 · HTML: https://mlsanity.com/paper/2104.04641/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 38 references that are in the map.
- First posted on arXiv 9 Apr 2021.

## Builds on (most cited first, 14 of 14)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [Learning to See in the Dark](https://mlsanity.com/paper/1805.01934/) (2018)
- [GA-Net: Guided Aggregation Net for End-to-end Stereo Matching](https://mlsanity.com/paper/1904.06587/) (2019)
- [DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch](https://mlsanity.com/paper/1909.05845/) (2019)
- [Deep Optics for Monocular Depth Estimation and 3D Object Detection](https://mlsanity.com/paper/1904.08601/) (2019)
- [Cascade Residual Learning: A Two-stage Convolutional Neural Network for Stereo Matching](https://mlsanity.com/paper/1708.09204/) (2017)
- [Deep Optics for Single-shot High-dynamic-range Imaging](https://mlsanity.com/paper/1908.00620/) (2019)
- [Handheld Mobile Photography in Very Low Light](https://mlsanity.com/paper/1910.11336/) (2019)
- [Learning Sensor Multiplexing Design through Back-propagation](https://mlsanity.com/paper/1605.07078/) (2016)
- [Polka Lines: Learning Structured Illumination and Reconstruction for Active Stereo](https://mlsanity.com/paper/2011.13117/) (2020)
- [MonSter: Awakening the Mono in Stereo](https://mlsanity.com/paper/1910.13708/) (2019)
- [A Learning-based Framework for Hybrid Depth-from-Defocus and Stereo Matching](https://mlsanity.com/paper/1708.00583/) (2017)

## Built on by (most cited first, 10 of 10)

- [A Differentiable Wave Optics Model for End-to-End Computational Imaging System Optimization](https://mlsanity.com/paper/2412.09774/) (2024)
- [Video Reconstruction from a Single Motion Blurred Image using Learned Dynamic Phase Coding](https://mlsanity.com/paper/2112.14768/) (2021)
- [Learning to See Through Dazzle](https://mlsanity.com/paper/2402.15919/) (2024)
- [Diffractive optical system design by cascaded propagation](https://mlsanity.com/paper/2202.11535/) (2022)
- [Collaborative On-Sensor Array Cameras](https://mlsanity.com/paper/2506.04061/) (2025)
- [Learned Off-aperture Encoding for Wide Field-of-view RGBD Imaging](https://mlsanity.com/paper/2507.22523/) (2025)
- [Neural Lithography: Close the Design-to-Manufacturing Gap in Computational Optics with a 'Real2Sim' Learned Photolithography Simulator](https://mlsanity.com/paper/2309.17343/) (2023)
- [Large-Area Fabrication-Aware Computational Diffractive Optics](https://mlsanity.com/paper/2505.22313/) (2025)
- [Learning to See Through Flare](https://mlsanity.com/paper/2508.13907/) (2025)
- [Depth from Dual Differential Defocus and Stereo Consensus](https://mlsanity.com/paper/2606.02906/) (2026)

## Abstract (arXiv)

Conventional stereo suffers from a fundamental trade-off between imaging volume and signal-to-noise ratio (SNR) -- due to the conflicting impact of aperture size on both these variables. Inspired by the extended depth of field cameras, we propose a novel end-to-end learning-based technique to overcome this limitation, by introducing a phase mask at the aperture plane of the cameras in a stereo imaging system. The phase mask creates a depth-dependent point spread function, allowing us to recover sharp image texture and stereo correspondence over a significantly extended depth of field (EDOF) than conventional stereo. The phase mask pattern, the EDOF image reconstruction, and the stereo disparity estimation are all trained together using an end-to-end learned deep neural network. We perform theoretical analysis and characterization of the proposed approach and show a 6x increase in volume that can be imaged in simulation. We also build an experimental prototype and validate the approach using real-world results acquired using this prototype system.