# Photonic convolutional neural network with pre-trained in situ training

arXiv: https://arxiv.org/abs/2604.02429 · HTML: https://mlsanity.com/paper/2604.02429/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 51 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Builds on (most cited first, 8 of 8)

- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Recent Trends in Deep Learning Based Natural Language Processing](https://mlsanity.com/paper/1708.02709/) (2017)
- [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://mlsanity.com/paper/2304.01433/) (2023)
- [All-Optical Machine Learning Using Diffractive Deep Neural Networks](https://mlsanity.com/paper/1804.08711/) (2018)
- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [Neuromorphic Silicon Photonic Networks](https://mlsanity.com/paper/1611.02272/) (2016)
- [Tunable Efficient Unitary Neural Networks (EUNN) and their application to RNNs](https://mlsanity.com/paper/1612.05231/) (2016)
- [Photonic tensor cores for machine learning](https://mlsanity.com/paper/2002.03780/) (2020)

## Abstract (arXiv)

Convolutional neural networks (CNNs) have transformed image processing, but the energy consumption and inference latency of electronic based implementations remain fundamental bottlenecks. These limitations have motivated the search for alternative hardware architectures beyond Complementary metal-oxide-semiconductor (CMOS) chips. Optical systems can perform linear matrix operations at the speed of light with extremely low energy dissipation, making them attractive for CNN acceleration. However, building a fully coherent photonic CNN that performs both linear and nonlinear operations and training it efficiently remains an open challenge. Here we present a fully photonic convolutional neural network (PCNN) that executes image classification in the optical domain, including convolution, max-pooling, nonlinear activation, and fully connected layers. The network achieves 94.49 percent accuracy on the MNIST dataset distributed across Mach Zehnder Interferometer (MZI) meshes, weighted Multimode Interferometer (MMI) trees, and a microring resonator based nonlinearity. A mathematically exact differentiable digital twin, enables backpropagation for ex situ pre training, reaches 97.45 percent digital accuracy. Trained phases are transferred one-to-one to the photonic hardware and refined via a gradient free algorithm that estimates the full gradient with only two forward passes. The architecture exhibits inherent robustness to non idealities, under the compound effect of propagation loss, MZI insertion loss, fabrication disorder, and thermal crosstalk. A bottom-up power analysis yields 10.83 W static chip consumption and 843 ns inference latency, translating to 220 to 330 times greater energy efficiency than state of the art electronic GPUs for single-image inference.