# Learned Off-aperture Encoding for Wide Field-of-view RGBD Imaging

arXiv: https://arxiv.org/abs/2507.22523 · HTML: https://mlsanity.com/paper/2507.22523/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 58 references that are in the map.
- First posted on arXiv 30 Jul 2025.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data](https://mlsanity.com/paper/2401.10891/) (2024)
- [A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation](https://mlsanity.com/paper/1512.02134/) (2015)
- [Simple Baselines for Image Restoration](https://mlsanity.com/paper/2204.04676/) (2022)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [Learning to See in the Dark](https://mlsanity.com/paper/1805.01934/) (2018)
- [Learning Single Camera Depth Estimation using Dual-Pixels](https://mlsanity.com/paper/1904.05822/) (2019)
- [Curriculum Learning for ab initio Deep Learned Refractive Optics](https://mlsanity.com/paper/2302.01089/) (2023)
- [Learning Sensor Multiplexing Design through Back-propagation](https://mlsanity.com/paper/1605.07078/) (2016)
- [Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array](https://mlsanity.com/paper/2006.08565/) (2020)
- [End-to-End Hybrid Refractive-Diffractive Lens Design with Differentiable Ray-Wave Model](https://mlsanity.com/paper/2406.00834/) (2024)
- [CodedStereo: Learned Phase Masks for Large Depth-of-field Stereo](https://mlsanity.com/paper/2104.04641/) (2021)
- [Hybrid Diffractive Optics Design via Hardware-in-the-Loop Methodology for Achromatic Extended-Depth-of-Field Imaging](https://mlsanity.com/paper/2203.16407/) (2022)
- [Depth Estimation and Image Restoration by Deep Learning from Defocused Images](https://mlsanity.com/paper/2302.10730/) (2023)
- [TiDy-PSFs: Computational Imaging with Time-Averaged Dynamic Point-Spread-Functions](https://mlsanity.com/paper/2303.17583/) (2023)
- [10-mega pixel snapshot compressive imaging with a hybrid coded aperture](https://mlsanity.com/paper/2106.15765/) (2021)
- [Diffractive optical system design by cascaded propagation](https://mlsanity.com/paper/2202.11535/) (2022)

## Built on by (most cited first, 1 of 1)

- [Large-FOV RGBD Imaging via Structured PSF Coding in a Bio-inspired Monocentric System](https://mlsanity.com/paper/2510.25314/) (2025)

## Abstract (arXiv)

End-to-end (E2E) designed imaging systems integrate coded optical designs with decoding algorithms to enhance imaging fidelity for diverse visual tasks. However, existing E2E designs encounter significant challenges in maintaining high image fidelity at wide fields of view, due to high computational complexity, as well as difficulties in modeling off-axis wave propagation while accounting for off-axis aberrations. In particular, the common approach of placing the encoding element into the aperture or pupil plane results in only a global control of the wavefront. To overcome these limitations, this work explores an additional design choice by positioning a DOE off-aperture, enabling a spatial unmixing of the degrees of freedom and providing local control over the wavefront over the image plane. Our approach further leverages hybrid refractive-diffractive optical systems by linking differentiable ray and wave optics modeling, thereby optimizing depth imaging quality and demonstrating system versatility. Experimental results reveal that the off-aperture DOE enhances the imaging quality by over 5 dB in PSNR at a FoV of approximately 45^circ when paired with a simple thin lens, outperforming traditional on-aperture systems. Furthermore, we successfully recover color and depth information at nearly 28^circ FoV using off-aperture DOE configurations with compound optics. Physical prototypes for both applications validate the effectiveness and versatility of the proposed method.