# A Differentiable Wave Optics Model for End-to-End Computational Imaging System Optimization

arXiv: https://arxiv.org/abs/2412.09774 · HTML: https://mlsanity.com/paper/2412.09774/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 13 Dec 2024.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks](https://mlsanity.com/paper/1710.11063/) (2017)
- [DiffuserCam: Lensless Single-exposure 3D Imaging](https://mlsanity.com/paper/1710.02134/) (2017)
- [Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics](https://mlsanity.com/paper/2009.00463/) (2020)
- [Physics-based Learned Design: Optimized Coded-Illumination for Quantitative Phase Imaging](https://mlsanity.com/paper/1808.03571/) (2018)
- [Optical Aberration Correction in Postprocessing using Imaging Simulation](https://mlsanity.com/paper/2305.05867/) (2023)
- [The Differentiable Lens: Compound Lens Search over Glass Surfaces and Materials for Object Detection](https://mlsanity.com/paper/2212.04441/) (2022)
- [Aberration-Aware Depth-from-Focus](https://mlsanity.com/paper/2303.04654/) (2023)
- [End-to-End Hybrid Refractive-Diffractive Lens Design with Differentiable Ray-Wave Model](https://mlsanity.com/paper/2406.00834/) (2024)
- [CodedStereo: Learned Phase Masks for Large Depth-of-field Stereo](https://mlsanity.com/paper/2104.04641/) (2021)
- [Task-Driven Lens Design](https://mlsanity.com/paper/2305.17185/) (2023)

## Built on by (most cited first, 6 of 6)

- [Automated design of compound lenses with discrete-continuous optimization](https://mlsanity.com/paper/2509.23572/) (2025)
- [Direct Kernel Optimization: Efficient Design for Opto-Electronic Convolutional Neural Networks](https://mlsanity.com/paper/2511.02065/) (2025)
- [Accelerating 4D Hyperspectral Imaging through Physics-Informed Neural Representation and Adaptive Sampling](https://mlsanity.com/paper/2604.06561/) (2026)
- [TurPy: a physics-based and differentiable optical turbulence simulator for algorithmic development and system optimization](https://mlsanity.com/paper/2604.07248/) (2026)
- [Low Latency Gaze Tracking via Latent Optical Sensing](https://mlsanity.com/paper/2605.17990/) (2026)
- [Realistic Compound-Lens Defocus Blur Synthesis](https://mlsanity.com/paper/2607.05837/) (2026)

## Abstract (arXiv)

End-to-end optimization, which simultaneously optimizes optics and algorithms, has emerged as a powerful data-driven method for computational imaging system design. This method achieves joint optimization through backpropagation by incorporating differentiable optics simulators to generate measurements and algorithms to extract information from measurements. However, due to high computational costs, it is challenging to model both aberration and diffraction in light transport for end-to-end optimization of compound optics. Therefore, most existing methods compromise physical accuracy by neglecting wave optics effects or off-axis aberrations, which raises concerns about the robustness of the resulting designs. In this paper, we propose a differentiable optics simulator that efficiently models both aberration and diffraction for compound optics. Using the simulator, we conduct end-to-end optimization on scene reconstruction and classification. Experimental results demonstrate that both lenses and algorithms adopt different configurations depending on whether wave optics is modeled. We also show that systems optimized without wave optics suffer from performance degradation when wave optics effects are introduced during testing. These findings underscore the importance of accurate wave optics modeling in optimizing imaging systems for robust, high-performance applications.