# End-to-End Hybrid Refractive-Diffractive Lens Design with Differentiable Ray-Wave Model

arXiv: https://arxiv.org/abs/2406.00834 · HTML: https://mlsanity.com/paper/2406.00834/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 68 references that are in the map.
- First posted on arXiv 2 Jun 2024.

## Builds on (most cited first, 13 of 13)

- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Simple Baselines for Image Restoration](https://mlsanity.com/paper/2204.04676/) (2022)
- [Deep Optics for Monocular Depth Estimation and 3D Object Detection](https://mlsanity.com/paper/1904.08601/) (2019)
- [Deep Optics for Single-shot High-dynamic-range Imaging](https://mlsanity.com/paper/1908.00620/) (2019)
- [Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics](https://mlsanity.com/paper/2009.00463/) (2020)
- [Curriculum Learning for ab initio Deep Learned Refractive Optics](https://mlsanity.com/paper/2302.01089/) (2023)
- [Optical Aberration Correction in Postprocessing using Imaging Simulation](https://mlsanity.com/paper/2305.05867/) (2023)
- [The Differentiable Lens: Compound Lens Search over Glass Surfaces and Materials for Object Detection](https://mlsanity.com/paper/2212.04441/) (2022)
- [Thin On-Sensor Nanophotonic Array Cameras](https://mlsanity.com/paper/2308.02797/) (2023)
- [Neural 'Etendue Expander for Ultra-Wide-Angle High-Fidelity Holographic Display](https://mlsanity.com/paper/2109.08123/) (2021)
- [Task-Driven Lens Design](https://mlsanity.com/paper/2305.17185/) (2023)
- [Hybrid Diffractive Optics Design via Hardware-in-the-Loop Methodology for Achromatic Extended-Depth-of-Field Imaging](https://mlsanity.com/paper/2203.16407/) (2022)
- [Diffractive lensless imaging with optimized Voronoi-Fresnel phase](https://mlsanity.com/paper/2109.13703/) (2021)

## Built on by (most cited first, 11 of 11)

- [A Differentiable Wave Optics Model for End-to-End Computational Imaging System Optimization](https://mlsanity.com/paper/2412.09774/) (2024)
- [Efficient Depth- and Spatially-Varying Image Simulation for Defocus Deblur](https://mlsanity.com/paper/2507.00372/) (2025)
- [Fovea Stacking: Imaging with Dynamic Localized Aberration Correction](https://mlsanity.com/paper/2506.00716/) (2025)
- [Collaborative On-Sensor Array Cameras](https://mlsanity.com/paper/2506.04061/) (2025)
- [Learned Off-aperture Encoding for Wide Field-of-view RGBD Imaging](https://mlsanity.com/paper/2507.22523/) (2025)
- [Towards Blind Lens Aberration Correction via Large LensLib Pre-training and Discrete Degradation Priors](https://mlsanity.com/paper/2511.17126/) (2025)
- [End-to-end differentiable design of geometric waveguide displays](https://mlsanity.com/paper/2601.04370/) (2026)
- [Computational metaoptics for imaging](https://mlsanity.com/paper/2411.09133/) (2024)
- [Large-Area Fabrication-Aware Computational Diffractive Optics](https://mlsanity.com/paper/2505.22313/) (2025)
- [A Differentiable Ray-Wave Framework for Hybrid Refractive-Diffractive System Modeling and Optimization](https://mlsanity.com/paper/2605.15418/) (2026)
- [Aperture-aware Dispersion 5-D Light-field Imaging Spectrometer](https://mlsanity.com/paper/2607.04635/) (2026)

## Abstract (arXiv)

Hybrid refractive-diffractive lenses combine the light efficiency of refractive lenses with the information encoding power of diffractive optical elements (DOE), showing great potential as the next generation of imaging systems. However, accurately simulating such hybrid designs is generally difficult, and in particular, there are no existing differentiable image formation models for hybrid lenses with sufficient accuracy. In this work, we propose a new hybrid ray-tracing and wave-propagation (ray-wave) model for accurate simulation of both optical aberrations and diffractive phase modulation, where the DOE is placed between the last refractive surface and the image sensor, i.e. away from the Fourier plane that is often used as a DOE position. The proposed ray-wave model is fully differentiable, enabling gradient back-propagation for end-to-end co-design of refractive-diffractive lens optimization and the image reconstruction network. We validate the accuracy of the proposed model by comparing the simulated point spread functions (PSFs) with theoretical results, as well as simulation experiments that show our model to be more accurate than solutions implemented in commercial software packages like Zemax. We demonstrate the effectiveness of the proposed model through real-world experiments and show significant improvements in both aberration correction and extended depth-of-field (EDoF) imaging. We believe the proposed model will motivate further investigation into a wide range of applications in computational imaging, computational photography, and advanced optical design. Code will be released upon publication.