# Spatially Varying Nanophotonic Neural Networks

arXiv: https://arxiv.org/abs/2308.03407 · HTML: https://mlsanity.com/paper/2308.03407/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 90 references that are in the map.
- First posted on arXiv 7 Aug 2023.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [All-Optical Machine Learning Using Diffractive Deep Neural Networks](https://mlsanity.com/paper/1804.08711/) (2018)
- [Compute Trends Across Three Eras of Machine Learning](https://mlsanity.com/paper/2202.05924/) (2022)
- [Photonics for artificial intelligence and neuromorphic computing](https://mlsanity.com/paper/2011.00111/) (2020)
- [Neuromorphic Silicon Photonic Networks](https://mlsanity.com/paper/1611.02272/) (2016)
- [Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit](https://mlsanity.com/paper/2008.11659/) (2020)
- [The Global Landscape of Neural Networks: An Overview](https://mlsanity.com/paper/2007.01429/) (2020)
- [Experimentally realized in situ backpropagation for deep learning in nanophotonic neural networks](https://mlsanity.com/paper/2205.08501/) (2022)
- [The physics of optical computing](https://mlsanity.com/paper/2308.00088/) (2023)
- [Analysis of Diffractive Optical Neural Networks and Their Integration with Electronic Neural Networks](https://mlsanity.com/paper/1810.01916/) (2018)
- [Image sensing with multilayer, nonlinear optical neural networks](https://mlsanity.com/paper/2207.14293/) (2022)
- [Ensemble learning of diffractive optical networks](https://mlsanity.com/paper/2009.06869/) (2020)
- [Scalable Optical Learning Operator](https://mlsanity.com/paper/2012.12404/) (2020)
- [An Optical Frontend for a Convolutional Neural Network](https://mlsanity.com/paper/1901.03661/) (2018)
- [Metasurface-Enabled On-Chip Multiplexed Diffractive Neural Networks in the Visible](https://mlsanity.com/paper/2107.07873/) (2021)
- [Class-specific Differential Detection in Diffractive Optical Neural Networks Improves Inference Accuracy](https://mlsanity.com/paper/1906.03417/) (2019)
- [Meta-optic Accelerators for Object Classifiers](https://mlsanity.com/paper/2201.11034/) (2022)
- [Digital Modeling on Large Kernel Metamaterial Neural Network](https://mlsanity.com/paper/2307.11862/) (2023)

## Built on by (most cited first, 12 of 12)

- [Compressed Meta-Optical Encoder for Image Classification](https://mlsanity.com/paper/2406.06534/) (2024)
- [Optical Diffusion Models for Image Generation](https://mlsanity.com/paper/2407.10897/) (2024)
- [Transferable polychromatic optical encoder for neural networks](https://mlsanity.com/paper/2411.02697/) (2024)
- [Neural Tangent Knowledge Distillation for Optical Convolutional Networks](https://mlsanity.com/paper/2508.08421/) (2025)
- [Latent Space Imaging](https://mlsanity.com/paper/2407.07052/) (2024)
- [SP2RINT: Spatially-Decoupled Physics-Inspired Progressive Inverse Optimization for Scalable, PDE-Constrained Meta-Optical Neural Network Training](https://mlsanity.com/paper/2505.18377/) (2025)
- [Toward Lifelong-Sustainable Electronic-Photonic AI Systems via Extreme Efficiency, Reconfigurability, and Robustness](https://mlsanity.com/paper/2509.07396/) (2025)
- [Metasurface-generated large and arbitrary analog convolution kernels for accelerated machine vision](https://mlsanity.com/paper/2409.18614/) (2024)
- [Computational metaoptics for imaging](https://mlsanity.com/paper/2411.09133/) (2024)
- [Physically Grounded Monocular Depth via Nanophotonic Wavefront Encoding](https://mlsanity.com/paper/2503.15770/) (2025)
- [Gradients of unitary optical neural networks using parameter-shift rule](https://mlsanity.com/paper/2506.11565/) (2025)
- [Direct Kernel Optimization: Efficient Design for Opto-Electronic Convolutional Neural Networks](https://mlsanity.com/paper/2511.02065/) (2025)

## Abstract (arXiv)

The explosive growth of computation and energy cost of artificial intelligence has spurred strong interests in new computing modalities as potential alternatives to conventional electronic processors. Photonic processors that execute operations using photons instead of electrons, have promised to enable optical neural networks with ultra-low latency and power consumption. However, existing optical neural networks, limited by the underlying network designs, have achieved image recognition accuracy far below that of state-of-the-art electronic neural networks. In this work, we close this gap by embedding massively parallelized optical computation into flat camera optics that perform neural network computation during the capture, before recording an image on the sensor. Specifically, we harness large kernels and propose a large-kernel spatially-varying convolutional neural network learned via low-dimensional reparameterization techniques. We experimentally instantiate the network with a flat meta-optical system that encompasses an array of nanophotonic structures designed to induce angle-dependent responses. Combined with an extremely lightweight electronic backend with approximately 2K parameters we demonstrate a reconfigurable nanophotonic neural network reaches 72.76% blind test classification accuracy on CIFAR-10 dataset, and, as such, the first time, an optical neural network outperforms the first modern digital neural network -- AlexNet (72.64%) with 57M parameters, bringing optical neural network into modern deep learning era.