# Collaborative On-Sensor Array Cameras

arXiv: https://arxiv.org/abs/2506.04061 · HTML: https://mlsanity.com/paper/2506.04061/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 102 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Builds on (most cited first, 19 of 19)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Toward Convolutional Blind Denoising of Real Photographs](https://mlsanity.com/paper/1807.04686/) (2018)
- [Deep Optics for Monocular Depth Estimation and 3D Object Detection](https://mlsanity.com/paper/1904.08601/) (2019)
- [Deep Optics for Single-shot High-dynamic-range Imaging](https://mlsanity.com/paper/1908.00620/) (2019)
- [DiffuserCam: Lensless Single-exposure 3D Imaging](https://mlsanity.com/paper/1710.02134/) (2017)
- [Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics](https://mlsanity.com/paper/2009.00463/) (2020)
- [Curriculum Learning for ab initio Deep Learned Refractive Optics](https://mlsanity.com/paper/2302.01089/) (2023)
- [FlatNet: Towards Photorealistic Scene Reconstruction from Lensless Measurements](https://mlsanity.com/paper/2010.15440/) (2020)
- [Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array](https://mlsanity.com/paper/2006.08565/) (2020)
- [Polka Lines: Learning Structured Illumination and Reconstruction for Active Stereo](https://mlsanity.com/paper/2011.13117/) (2020)
- [DeepBinaryMask: Learning a Binary Mask for Video Compressive Sensing](https://mlsanity.com/paper/1607.03343/) (2016)
- [Fast Two-step Blind Optical Aberration Correction](https://mlsanity.com/paper/2208.00950/) (2022)
- [Convolutional neural networks that teach microscopes how to image](https://mlsanity.com/paper/1709.07223/) (2017)
- [Thin On-Sensor Nanophotonic Array Cameras](https://mlsanity.com/paper/2308.02797/) (2023)
- [End-to-End Hybrid Refractive-Diffractive Lens Design with Differentiable Ray-Wave Model](https://mlsanity.com/paper/2406.00834/) (2024)
- [CodedStereo: Learned Phase Masks for Large Depth-of-field Stereo](https://mlsanity.com/paper/2104.04641/) (2021)
- [Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in Time-of-Flight Imaging](https://mlsanity.com/paper/2103.16693/) (2021)

## Built on by (most cited first, 1 of 1)

- [Low Latency Gaze Tracking via Latent Optical Sensing](https://mlsanity.com/paper/2605.17990/) (2026)

## Abstract (arXiv)

Modern nanofabrication techniques have enabled us to manipulate the wavefront of light with sub-wavelength-scale structures, offering the potential to replace bulky refractive surfaces in conventional optics with ultrathin metasurfaces. In theory, arrays of nanoposts provide unprecedented control over manipulating the wavefront in terms of phase, polarization, and amplitude at the nanometer resolution. A line of recent work successfully investigates flat computational cameras that replace compound lenses with a single metalens or an array of metasurfaces a few millimeters from the sensor. However, due to the inherent wavelength dependence of metalenses, in practice, these cameras do not match their refractive counterparts in image quality for broadband imaging, and may even suffer from hallucinations when relying on generative reconstruction methods. In this work, we investigate a collaborative array of metasurface elements that are jointly learned to perform broadband imaging. To this end, we learn a nanophotonics array with 100-million nanoposts that is end-to-end jointly optimized over the full visible spectrum--a design task that existing inverse design methods or learning approaches cannot support due to memory and compute limitations. We introduce a distributed meta-optics learning method to tackle this challenge. This allows us to optimize a large parameter array along with a learned meta-atom proxy and a non-generative reconstruction method that is parallax-aware and noise-aware. The proposed camera performs favorably in simulation and in all experimental tests irrespective of the scene illumination spectrum.