# Thin On-Sensor Nanophotonic Array Cameras

arXiv: https://arxiv.org/abs/2308.02797 · HTML: https://mlsanity.com/paper/2308.02797/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 96 references that are in the map.
- First posted on arXiv 5 Aug 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Denoising Diffusion Restoration Models](https://mlsanity.com/paper/2201.11793/) (2022)
- [The Little Engine that Could: Regularization by Denoising (RED)](https://mlsanity.com/paper/1611.02862/) (2016)
- [Diffusion-SDF: Conditional Generative Modeling of Signed Distance Functions](https://mlsanity.com/paper/2211.13757/) (2022)
- [Zoom To Learn, Learn To Zoom](https://mlsanity.com/paper/1905.05169/) (2019)
- [Deep Optics for Monocular Depth Estimation and 3D Object Detection](https://mlsanity.com/paper/1904.08601/) (2019)
- [Deep Optics for Single-shot High-dynamic-range Imaging](https://mlsanity.com/paper/1908.00620/) (2019)
- [DiffuserCam: Lensless Single-exposure 3D Imaging](https://mlsanity.com/paper/1710.02134/) (2017)
- [Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics](https://mlsanity.com/paper/2009.00463/) (2020)
- [Learning Sensor Multiplexing Design through Back-propagation](https://mlsanity.com/paper/1605.07078/) (2016)
- [FlatNet: Towards Photorealistic Scene Reconstruction from Lensless Measurements](https://mlsanity.com/paper/2010.15440/) (2020)
- [Learning Depth from Single Images with Deep Neural Network Embedding Focal Length](https://mlsanity.com/paper/1803.10039/) (2018)
- [Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array](https://mlsanity.com/paper/2006.08565/) (2020)
- [Polka Lines: Learning Structured Illumination and Reconstruction for Active Stereo](https://mlsanity.com/paper/2011.13117/) (2020)
- [Universal and Flexible Optical Aberration Correction Using Deep-Prior Based Deconvolution](https://mlsanity.com/paper/2104.03078/) (2021)
- [Data-Driven Design for Fourier Ptychographic Microscopy](https://mlsanity.com/paper/1904.04175/) (2019)
- [On Maximum-a-Posteriori estimation with Plug & Play priors and stochastic gradient descent](https://mlsanity.com/paper/2201.06133/) (2022)
- [Convolutional neural networks that teach microscopes how to image](https://mlsanity.com/paper/1709.07223/) (2017)
- [DeepToF: Off-the-Shelf Real-Time Correction of Multipath Interference in Time-of-Flight Imaging](https://mlsanity.com/paper/1805.09305/) (2018)
- [Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging](https://mlsanity.com/paper/2103.16693/) (2021)
- [Unrolled Primal-Dual Networks for Lensless Cameras](https://mlsanity.com/paper/2203.04353/) (2022)

## Built on by (most cited first, 13 of 13)

- [End-to-End Hybrid Refractive-Diffractive Lens Design with Differentiable Ray-Wave Model](https://mlsanity.com/paper/2406.00834/) (2024)
- [Imaging for All-Day Wearable Smart Glasses](https://mlsanity.com/paper/2504.13060/) (2025)
- [UniCoRN: Latent Diffusion-based Unified Controllable Image Restoration Network across Multiple Degradations](https://mlsanity.com/paper/2503.15868/) (2025)
- [Degradation-Modeled Multipath Diffusion for Tunable Metalens Photography](https://mlsanity.com/paper/2506.22753/) (2025)
- [Enabling High-Quality In-the-Wild Imaging from Severely Aberrated Metalens Bursts](https://mlsanity.com/paper/2510.10083/) (2025)
- [Aberration Correcting Vision Transformers for High-Fidelity Metalens Imaging](https://mlsanity.com/paper/2412.04591/) (2024)
- [Collaborative On-Sensor Array Cameras](https://mlsanity.com/paper/2506.04061/) (2025)
- [Self-attention-based non-linear basis transformations for compact latent space modelling of dynamic optical fibre transmission matrices](https://mlsanity.com/paper/2406.07775/) (2024)
- [Paleoinspired Vision: From Exploring Colour Vision Evolution to Inspiring Camera Design](https://mlsanity.com/paper/2412.19439/) (2024)
- [Physically Grounded Monocular Depth via Nanophotonic Wavefront Encoding](https://mlsanity.com/paper/2503.15770/) (2025)
- [Large-Area Fabrication-Aware Computational Diffractive Optics](https://mlsanity.com/paper/2505.22313/) (2025)
- [Machine-Learning-Assisted Photonic Device Development: A Multiscale Approach from Theory to Characterization](https://mlsanity.com/paper/2506.20056/) (2025)
- [Diffusion Algorithm for Metalens Optical Aberration Correction](https://mlsanity.com/paper/2511.12689/) (2025)

## Abstract (arXiv)

Today's commodity camera systems rely on compound optics to map light originating from the scene to positions on the sensor where it gets recorded as an image. To record images without optical aberrations, i.e., deviations from Gauss' linear model of optics, typical lens systems introduce increasingly complex stacks of optical elements which are responsible for the height of existing commodity cameras. In this work, we investigate flat nanophotonic computational cameras as an alternative that employs an array of skewed lenslets and a learned reconstruction approach. The optical array is embedded on a metasurface that, at 700~nm height, is flat and sits on the sensor cover glass at 2.5~mm focal distance from the sensor. To tackle the highly chromatic response of a metasurface and design the array over the entire sensor, we propose a differentiable optimization method that continuously samples over the visible spectrum and factorizes the optical modulation for different incident fields into individual lenses. We reconstruct a megapixel image from our flat imager with a learned probabilistic reconstruction method that employs a generative diffusion model to sample an implicit prior. To tackle scene-dependent aberrations in broadband, we propose a method for acquiring paired captured training data in varying illumination conditions. We assess the proposed flat camera design in simulation and with an experimental prototype, validating that the method is capable of recovering images from diverse scenes in broadband with a single nanophotonic layer.