# Metasurface-generated large and arbitrary analog convolution kernels for accelerated machine vision

arXiv: https://arxiv.org/abs/2409.18614 · HTML: https://mlsanity.com/paper/2409.18614/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 57 references that are in the map.
- First posted on arXiv 27 Sep 2024.

## Builds on (most cited first, 18 of 18)

- [All-Optical Machine Learning Using Diffractive Deep Neural Networks](https://mlsanity.com/paper/1804.08711/) (2018)
- [Photonics for artificial intelligence and neuromorphic computing](https://mlsanity.com/paper/2011.00111/) (2020)
- [Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit](https://mlsanity.com/paper/2008.11659/) (2020)
- [Image sensing with multilayer, nonlinear optical neural networks](https://mlsanity.com/paper/2207.14293/) (2022)
- [Massively Parallel Amplitude-Only Fourier Neural Network](https://mlsanity.com/paper/2008.05853/) (2020)
- [Nonlinear Processing with Linear Optics](https://mlsanity.com/paper/2307.08533/) (2023)
- [An Optical Frontend for a Convolutional Neural Network](https://mlsanity.com/paper/1901.03661/) (2018)
- [Metasurface-Enabled On-Chip Multiplexed Diffractive Neural Networks in the Visible](https://mlsanity.com/paper/2107.07873/) (2021)
- [Universal Linear Intensity Transformations Using Spatially-Incoherent Diffractive Processors](https://mlsanity.com/paper/2303.13037/) (2023)
- [Meta-optic Accelerators for Object Classifiers](https://mlsanity.com/paper/2201.11034/) (2022)
- [Single-Shot Optical Neural Network](https://mlsanity.com/paper/2205.09103/) (2022)
- [Spatially Varying Nanophotonic Neural Networks](https://mlsanity.com/paper/2308.03407/) (2023)
- [Object Detection in Hyperspectral Image via Unified Spectral-Spatial Feature Aggregation](https://mlsanity.com/paper/2306.08370/) (2023)
- [Photonic Advantage of Optical Encoders](https://mlsanity.com/paper/2305.01743/) (2023)
- [ASP Vision: Optically Computing the First Layer of Convolutional Neural Networks using Angle Sensitive Pixels](https://mlsanity.com/paper/1605.03621/) (2016)
- [Digital Modeling on Large Kernel Metamaterial Neural Network](https://mlsanity.com/paper/2307.11862/) (2023)
- [ExtremeMETA: High-speed Lightweight Image Segmentation Model by Remodeling Multi-channel Metamaterial Imagers](https://mlsanity.com/paper/2405.17568/) (2024)
- [Integration of Programmable Diffraction with Digital Neural Networks](https://mlsanity.com/paper/2406.10688/) (2024)

## Abstract (arXiv)

In the rapidly evolving field of artificial intelligence, convolutional neural networks are essential for tackling complex challenges such as machine vision and medical diagnosis. Recently, to address the challenges in processing speed and power consumption of conventional digital convolution operations, many optical components have been suggested to replace the digital convolution layer in the neural network, accelerating various machine vision tasks. Nonetheless, the analog nature of the optical convolution kernel has not been fully explored. Here, we develop a spatial frequency domain training method to create arbitrarily shaped analog convolution kernels using an optical metasurface as the convolution layer, with its receptive field largely surpassing digital convolution kernels. By employing spatial multiplexing, the multiple parallel convolution kernels with both positive and negative weights are generated under the incoherent illumination condition. We experimentally demonstrate a 98.59% classification accuracy on the MNIST dataset, with simulations showing 92.63% and 68.67% accuracy on the Fashion-MNIST and CIFAR-10 datasets with additional digital layers. This work underscores the unique advantage of analog optical convolution, offering a promising avenue to accelerate machine vision tasks, especially in edge devices.