# Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics

arXiv: https://arxiv.org/abs/2009.00463 · HTML: https://mlsanity.com/paper/2009.00463/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Sep 2020.

## Built on by (most cited first, 30 of 39)

- [Curriculum Learning for ab initio Deep Learned Refractive Optics](https://mlsanity.com/paper/2302.01089/) (2023)
- [Image sensing with multilayer, nonlinear optical neural networks](https://mlsanity.com/paper/2207.14293/) (2022)
- [Thin On-Sensor Nanophotonic Array Cameras](https://mlsanity.com/paper/2308.02797/) (2023)
- [Aberration-Aware Depth-from-Focus](https://mlsanity.com/paper/2303.04654/) (2023)
- [End-to-End Hybrid Refractive-Diffractive Lens Design with Differentiable Ray-Wave Model](https://mlsanity.com/paper/2406.00834/) (2024)
- [Task-Driven Lens Design](https://mlsanity.com/paper/2305.17185/) (2023)
- [Exploring Quasi-Global Solutions to Compound Lens Based Computational Imaging Systems](https://mlsanity.com/paper/2404.19201/) (2024)
- [CodedEvents: Optimal Point-Spread-Function Engineering for 3D-Tracking with Event Cameras](https://mlsanity.com/paper/2406.09409/) (2024)
- [Revealing the preference for correcting separated aberrations in joint optic-image design](https://mlsanity.com/paper/2309.04342/) (2023)
- [Learning Lens Blur Fields](https://mlsanity.com/paper/2310.11535/) (2023)
- [A Differentiable Wave Optics Model for End-to-End Computational Imaging System Optimization](https://mlsanity.com/paper/2412.09774/) (2024)
- [Limitations of Data-Driven Spectral Reconstruction -- An Optics-Aware Analysis](https://mlsanity.com/paper/2401.03835/) (2024)
- [Deep Hyperspectral-Depth Reconstruction Using Single Color-Dot Projection](https://mlsanity.com/paper/2204.03929/) (2022)
- [Analysis of Diffractive Neural Networks for Seeing Through Random Diffusers](https://mlsanity.com/paper/2205.00428/) (2022)
- [Dispersed Structured Light for Hyperspectral 3D Imaging](https://mlsanity.com/paper/2311.18287/) (2023)
- [Polarization Multi-Image Synthesis with Birefringent Metasurfaces](https://mlsanity.com/paper/2307.08106/) (2023)
- [Depth Estimation from a Single Optical Encoded Image using a Learned Colored-Coded Aperture](https://mlsanity.com/paper/2309.08033/) (2023)
- [Learning to See Through Dazzle](https://mlsanity.com/paper/2402.15919/) (2024)
- [Dense Dispersed Structured Light for Hyperspectral 3D Imaging of Dynamic Scenes](https://mlsanity.com/paper/2412.01140/) (2024)
- [Event Spectroscopy: Event-based Multispectral and Depth Sensing using Structured Light](https://mlsanity.com/paper/2509.06741/) (2025)
- [Autoregressive High-Order Finite Difference Modulo Imaging: High-Dynamic Range for Computer Vision Applications](https://mlsanity.com/paper/2504.04228/) (2025)
- [Diffractive optical system design by cascaded propagation](https://mlsanity.com/paper/2202.11535/) (2022)
- [Foveated Thermal Computational Imaging in the Wild Using All-Silicon Meta-Optics](https://mlsanity.com/paper/2212.06345/) (2022)
- [Integration of Programmable Diffraction with Digital Neural Networks](https://mlsanity.com/paper/2406.10688/) (2024)
- [Collaborative On-Sensor Array Cameras](https://mlsanity.com/paper/2506.04061/) (2025)
- [Neural Lithography: Close the Design-to-Manufacturing Gap in Computational Optics with a 'Real2Sim' Learned Photolithography Simulator](https://mlsanity.com/paper/2309.17343/) (2023)
- [Computational Spectral Imaging with Unified Encoding Model: A Comparative Study and Beyond](https://mlsanity.com/paper/2312.13310/) (2023)
- [Solving Vision Tasks with Simple Photoreceptors Instead of Cameras](https://mlsanity.com/paper/2406.11769/) (2024)
- [Spectrum from Defocus: Fast Spectral Imaging with Chromatic Focal Stack](https://mlsanity.com/paper/2503.20184/) (2025)
- [Large-Area Fabrication-Aware Computational Diffractive Optics](https://mlsanity.com/paper/2505.22313/) (2025)

## Abstract (arXiv)

Imaging depth and spectrum have been extensively studied in isolation from each other for decades. Recently, hyperspectral-depth (HS-D) imaging emerges to capture both information simultaneously by combining two different imaging systems; one for depth, the other for spectrum. While being accurate, this combinational approach induces increased form factor, cost, capture time, and alignment/registration problems. In this work, departing from the combinational principle, we propose a compact single-shot monocular HS-D imaging method. Our method uses a diffractive optical element (DOE), the point spread function of which changes with respect to both depth and spectrum. This enables us to reconstruct spectrum and depth from a single captured image. To this end, we develop a differentiable simulator and a neural-network-based reconstruction that are jointly optimized via automatic differentiation. To facilitate learning the DOE, we present a first HS-D dataset by building a benchtop HS-D imager that acquires high-quality ground truth. We evaluate our method with synthetic and real experiments by building an experimental prototype and achieve state-of-the-art HS-D imaging results.