# Image sensing with multilayer, nonlinear optical neural networks

arXiv: https://arxiv.org/abs/2207.14293 · HTML: https://mlsanity.com/paper/2207.14293/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 93 references that are in the map.
- First posted on arXiv 27 Jul 2022.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [All-Optical Machine Learning Using Diffractive Deep Neural Networks](https://mlsanity.com/paper/1804.08711/) (2018)
- [Photonics for artificial intelligence and neuromorphic computing](https://mlsanity.com/paper/2011.00111/) (2020)
- [Why does deep and cheap learning work so well?](https://mlsanity.com/paper/1608.08225/) (2016)
- [Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit](https://mlsanity.com/paper/2008.11659/) (2020)
- [Lensless computational imaging through deep learning](https://mlsanity.com/paper/1702.08516/) (2017)
- [Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics](https://mlsanity.com/paper/2009.00463/) (2020)
- [An optical neural network using less than 1 photon per multiplication](https://mlsanity.com/paper/2104.13467/) (2021)
- [An Optical Frontend for a Convolutional Neural Network](https://mlsanity.com/paper/1901.03661/) (2018)
- [Metasurface-Enabled On-Chip Multiplexed Diffractive Neural Networks in the Visible](https://mlsanity.com/paper/2107.07873/) (2021)
- [Time-Multiplexed Coded Aperture Imaging: Learned Coded Aperture and Pixel Exposures for Compressive Imaging Systems](https://mlsanity.com/paper/2104.02820/) (2021)
- [Meta-optic Accelerators for Object Classifiers](https://mlsanity.com/paper/2201.11034/) (2022)
- [Single-Shot Optical Neural Network](https://mlsanity.com/paper/2205.09103/) (2022)
- [Deep Cytometry: Deep learning with Real-time Inference in Cell Sorting and Flow Cytometry](https://mlsanity.com/paper/1904.09233/) (2019)
- [Real-time Hyperspectral Imaging in Hardware via Trained Metasurface Encoders](https://mlsanity.com/paper/2204.02084/) (2022)
- [Multi-element microscope optimization by a learned sensing network with composite physical layers](https://mlsanity.com/paper/2006.15404/) (2020)
- [Learning rich optical embeddings for privacy-preserving lensless image classification](https://mlsanity.com/paper/2206.01429/) (2022)

## Built on by (most cited first, 25 of 25)

- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [Nonlinear optical encoding enabled by recurrent linear scattering](https://mlsanity.com/paper/2307.08558/) (2023)
- [Spatially Varying Nanophotonic Neural Networks](https://mlsanity.com/paper/2308.03407/) (2023)
- [Photonic Advantage of Optical Encoders](https://mlsanity.com/paper/2305.01743/) (2023)
- [Optical Generative Models](https://mlsanity.com/paper/2410.17970/) (2024)
- [All-optical image denoising using a diffractive visual processor](https://mlsanity.com/paper/2309.09215/) (2023)
- [Compressed Meta-Optical Encoder for Image Classification](https://mlsanity.com/paper/2406.06534/) (2024)
- [Transferable polychromatic optical encoder for neural networks](https://mlsanity.com/paper/2411.02697/) (2024)
- [Neural Network Methods for Radiation Detectors and Imaging](https://mlsanity.com/paper/2311.05726/) (2023)
- [Neural Tangent Knowledge Distillation for Optical Convolutional Networks](https://mlsanity.com/paper/2508.08421/) (2025)
- [Photonics-Enhanced Graph Convolutional Networks](https://mlsanity.com/paper/2512.15549/) (2025)
- [Quantum-secure multiparty deep learning](https://mlsanity.com/paper/2408.05629/) (2024)
- [Experimental verification of the quantum nature of a neural network](https://mlsanity.com/paper/2209.07577/) (2022)
- [Intelligent Multi-channel Meta-imagers for Accelerating Machine Vision](https://mlsanity.com/paper/2306.07365/) (2023)
- [Efficient option pricing with unary-based photonic computing chip and generative adversarial learning](https://mlsanity.com/paper/2308.04493/) (2023)
- [Compute-first optical detection for noise-resilient visual perception](https://mlsanity.com/paper/2403.09612/) (2024)
- [Metasurface-generated large and arbitrary analog convolution kernels for accelerated machine vision](https://mlsanity.com/paper/2409.18614/) (2024)
- [Programmable metasurfaces for future photonic artificial intelligence](https://mlsanity.com/paper/2505.11659/) (2025)
- [Machine Intelligence on Wireless Edge Networks](https://mlsanity.com/paper/2506.12210/) (2025)
- [Spatially Parallel All-optical Neural Networks](https://mlsanity.com/paper/2509.23611/) (2025)
- [Optical diffraction neural networks assisted computational ghost imaging through dynamic scattering media](https://mlsanity.com/paper/2511.22913/) (2025)
- [Spatio-Temporal Difference Guided Motion Deblurring with the Complementary Vision Sensor](https://mlsanity.com/paper/2604.10554/) (2026)
- [Harnessing Photonics for Machine Intelligence](https://mlsanity.com/paper/2604.10841/) (2026)
- [Low Latency Gaze Tracking via Latent Optical Sensing](https://mlsanity.com/paper/2605.17990/) (2026)
- [Generative modelling powered by room-temperature polariton condensates](https://mlsanity.com/paper/2606.15344/) (2026)

## Abstract (arXiv)

Optical imaging is commonly used for both scientific and technological applications across industry and academia. In image sensing, a measurement, such as of an object's position, is performed by computational analysis of a digitized image. An emerging image-sensing paradigm breaks this delineation between data collection and analysis by designing optical components to perform not imaging, but encoding. By optically encoding images into a compressed, low-dimensional latent space suitable for efficient post-analysis, these image sensors can operate with fewer pixels and fewer photons, allowing higher-throughput, lower-latency operation. Optical neural networks (ONNs) offer a platform for processing data in the analog, optical domain. ONN-based sensors have however been limited to linear processing, but nonlinearity is a prerequisite for depth, and multilayer NNs significantly outperform shallow NNs on many tasks. Here, we realize a multilayer ONN pre-processor for image sensing, using a commercial image intensifier as a parallel optoelectronic, optical-to-optical nonlinear activation function. We demonstrate that the nonlinear ONN pre-processor can achieve compression ratios of up to 800:1 while still enabling high accuracy across several representative computer-vision tasks, including machine-vision benchmarks, flow-cytometry image classification, and identification of objects in real scenes. In all cases we find that the ONN's nonlinearity and depth allowed it to outperform a purely linear ONN encoder. Although our experiments are specialized to ONN sensors for incoherent-light images, alternative ONN platforms should facilitate a range of ONN sensors. These ONN sensors may surpass conventional sensors by pre-processing optical information in spatial, temporal, and/or spectral dimensions, potentially with coherent and quantum qualities, all natively in the optical domain.