# Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit

arXiv: https://arxiv.org/abs/2008.11659 · HTML: https://mlsanity.com/paper/2008.11659/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Aug 2020.

## Built on by (most cited first, 30 of 65)

- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [An optical neural network using less than 1 photon per multiplication](https://mlsanity.com/paper/2104.13467/) (2021)
- [Image sensing with multilayer, nonlinear optical neural networks](https://mlsanity.com/paper/2207.14293/) (2022)
- [Nonlinear Processing with Linear Optics](https://mlsanity.com/paper/2307.08533/) (2023)
- [Forward-Forward Training of an Optical Neural Network](https://mlsanity.com/paper/2305.19170/) (2023)
- [Hybrid training of optical neural networks](https://mlsanity.com/paper/2203.11207/) (2022)
- [A compact butterfly-style silicon photonic-electronic neural chip for hardware-efficient deep learning](https://mlsanity.com/paper/2111.06705/) (2021)
- [Unidirectional Imaging using Deep Learning-Designed Materials](https://mlsanity.com/paper/2212.02025/) (2022)
- [Single-Shot Optical Neural Network](https://mlsanity.com/paper/2205.09103/) (2022)
- [Spatially Varying Nanophotonic Neural Networks](https://mlsanity.com/paper/2308.03407/) (2023)
- [Polarization Multiplexed Diffractive Computing: All-Optical Implementation of a Group of Linear Transformations Through a Polarization-Encoded Diffractive Network](https://mlsanity.com/paper/2203.13482/) (2022)
- [Optical Transformers](https://mlsanity.com/paper/2302.10360/) (2023)
- [Dual adaptive training of photonic neural networks](https://mlsanity.com/paper/2212.06141/) (2022)
- [Multiplexed gradient descent: Fast online training of modern datasets on hardware neural networks without backpropagation](https://mlsanity.com/paper/2303.03986/) (2023)
- [Quantum-limited stochastic optical neural networks operating at a few quanta per activation](https://mlsanity.com/paper/2307.15712/) (2023)
- [Optical Generative Models](https://mlsanity.com/paper/2410.17970/) (2024)
- [Training neural networks with end-to-end optical backpropagation](https://mlsanity.com/paper/2308.05226/) (2023)
- [All-optical graph representation learning using integrated diffractive photonic computing units](https://mlsanity.com/paper/2204.10978/) (2022)
- [Compressed Meta-Optical Encoder for Image Classification](https://mlsanity.com/paper/2406.06534/) (2024)
- [Classification and reconstruction of spatially overlapping phase images using diffractive optical networks](https://mlsanity.com/paper/2108.07977/) (2021)
- [Skydiver: A Spiking Neural Network Accelerator Exploiting Spatio-Temporal Workload Balance](https://mlsanity.com/paper/2203.07516/) (2022)
- [Photonic Reconfigurable Accelerators for Efficient Inference of CNNs with Mixed-Sized Tensors](https://mlsanity.com/paper/2207.05278/) (2022)
- [All-optical image classification through unknown random diffusers using a single-pixel diffractive network](https://mlsanity.com/paper/2208.03968/) (2022)
- [Physics-aware Differentiable Discrete Codesign for Diffractive Optical Neural Networks](https://mlsanity.com/paper/2209.14252/) (2022)
- [Optical multi-task learning using multi-wavelength diffractive deep neural networks](https://mlsanity.com/paper/2212.00022/) (2022)
- [M3ICRO: Machine Learning-Enabled Compact Photonic Tensor Core based on PRogrammable Multi-Operand Multimode Interference](https://mlsanity.com/paper/2305.19505/) (2023)
- [Time-lapse image classification using a diffractive neural network](https://mlsanity.com/paper/2208.10802/) (2022)
- [SCONNA: A Stochastic Computing Based Optical Accelerator for Ultra-Fast, Energy-Efficient Inference of Integer-Quantized CNNs](https://mlsanity.com/paper/2302.07036/) (2023)
- [Optical Diffusion Models for Image Generation](https://mlsanity.com/paper/2407.10897/) (2024)
- [Asymmetrical estimator for training encapsulated deep photonic neural networks](https://mlsanity.com/paper/2405.18458/) (2024)

## Abstract (arXiv)

Application-specific optical processors have been considered disruptive technologies for modern computing that can fundamentally accelerate the development of artificial intelligence (AI) by offering substantially improved computing performance. Recent advancements in optical neural network architectures for neural information processing have been applied to perform various machine learning tasks. However, the existing architectures have limited complexity and performance; and each of them requires its own dedicated design that cannot be reconfigured to switch between different neural network models for different applications after deployment. Here, we propose an optoelectronic reconfigurable computing paradigm by constructing a diffractive processing unit (DPU) that can efficiently support different neural networks and achieve a high model complexity with millions of neurons. It allocates almost all of its computational operations optically and achieves extremely high speed of data modulation and large-scale network parameter updating by dynamically programming optical modulators and photodetectors. We demonstrated the reconfiguration of the DPU to implement various diffractive feedforward and recurrent neural networks and developed a novel adaptive training approach to circumvent the system imperfections. We applied the trained networks for high-speed classifying of handwritten digit images and human action videos over benchmark datasets, and the experimental results revealed a comparable classification accuracy to the electronic computing approaches. Furthermore, our prototype system built with off-the-shelf optoelectronic components surpasses the performance of state-of-the-art graphics processing units (GPUs) by several times on computing speed and more than an order of magnitude on system energy efficiency.