# Integration of Programmable Diffraction with Digital Neural Networks

arXiv: https://arxiv.org/abs/2406.10688 · HTML: https://mlsanity.com/paper/2406.10688/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 171 references that are in the map.
- First posted on arXiv 15 Jun 2024.

## Builds on (most cited first, 30 of 58)

- [All-Optical Machine Learning Using Diffractive Deep Neural Networks](https://mlsanity.com/paper/1804.08711/) (2018)
- [Universality of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1805.10769/) (2018)
- [Art and the science of generative AI: A deeper dive](https://mlsanity.com/paper/2306.04141/) (2023)
- [Neuromorphic Silicon Photonic Networks](https://mlsanity.com/paper/1611.02272/) (2016)
- [Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit](https://mlsanity.com/paper/2008.11659/) (2020)
- [Deep neural networks for the evaluation and design of photonic devices](https://mlsanity.com/paper/2007.00084/) (2020)
- [Training of photonic neural networks through in situ backpropagation](https://mlsanity.com/paper/1805.09943/) (2018)
- [Nonlinear Approximation and (Deep) ReLU Networks](https://mlsanity.com/paper/1905.02199/) (2019)
- [Deep Learning-enabled Virtual Histological Staining of Biological Samples](https://mlsanity.com/paper/2211.06822/) (2022)
- [Single-shot Hyperspectral-Depth Imaging with Learned Diffractive Optics](https://mlsanity.com/paper/2009.00463/) (2020)
- [Misalignment Resilient Diffractive Optical Networks](https://mlsanity.com/paper/2005.11464/) (2020)
- [Analysis of Diffractive Optical Neural Networks and Their Integration with Electronic Neural Networks](https://mlsanity.com/paper/1810.01916/) (2018)
- [A Generative Model for Inverse Design of Metamaterials](https://mlsanity.com/paper/1805.10181/) (2018)
- [Design of Task-Specific Optical Systems Using Broadband Diffractive Neural Networks](https://mlsanity.com/paper/1909.06553/) (2019)
- [All-Optical Synthesis of an Arbitrary Linear Transformation Using Diffractive Surfaces](https://mlsanity.com/paper/2108.09833/) (2021)
- [Rapid identification of pathogenic bacteria using Raman spectroscopy and deep learning](https://mlsanity.com/paper/1901.07666/) (2019)
- [Digital Electronics and Analog Photonics for Convolutional Neural Networks (DEAP-CNNs)](https://mlsanity.com/paper/1907.01525/) (2019)
- [Diffractive all-optical computing for quantitative phase imaging](https://mlsanity.com/paper/2201.08964/) (2022)
- [To image, or not to image: Class-specific diffractive cameras with all-optical erasure of undesired objects](https://mlsanity.com/paper/2205.13122/) (2022)
- [DeepBinaryMask: Learning a Binary Mask for Video Compressive Sensing](https://mlsanity.com/paper/1607.03343/) (2016)
- [Computational Imaging Without a Computer: Seeing Through Random Diffusers at the Speed of Light](https://mlsanity.com/paper/2107.06586/) (2021)
- [Physics-based Learned Design: Optimized Coded-Illumination for Quantitative Phase Imaging](https://mlsanity.com/paper/1808.03571/) (2018)
- [Data-Driven Design for Fourier Ptychographic Microscopy](https://mlsanity.com/paper/1904.04175/) (2019)
- [Universal Linear Intensity Transformations Using Spatially-Incoherent Diffractive Processors](https://mlsanity.com/paper/2303.13037/) (2023)
- [Designing nanophotonic structures using conditional-deep convolutional generative adversarial networks](https://mlsanity.com/paper/1903.08432/) (2019)
- [Snapshot Multispectral Imaging Using a Diffractive Optical Network](https://mlsanity.com/paper/2212.05217/) (2022)
- [Data class-specific all-optical transformations and encryption](https://mlsanity.com/paper/2212.12873/) (2022)
- [Freeform Diffractive Metagrating Design Based on Generative Adversarial Networks](https://mlsanity.com/paper/1811.12436/) (2018)
- [Meta-optic Accelerators for Object Classifiers](https://mlsanity.com/paper/2201.11034/) (2022)
- [Super-resolution image display using diffractive decoders](https://mlsanity.com/paper/2206.07281/) (2022)

## Built on by (most cited first, 1 of 1)

- [Metasurface-generated large and arbitrary analog convolution kernels for accelerated machine vision](https://mlsanity.com/paper/2409.18614/) (2024)

## Abstract (arXiv)

Optical imaging and sensing systems based on diffractive elements have seen massive advances over the last several decades. Earlier generations of diffractive optical processors were, in general, designed to deliver information to an independent system that was separately optimized, primarily driven by human vision or perception. With the recent advances in deep learning and digital neural networks, there have been efforts to establish diffractive processors that are jointly optimized with digital neural networks serving as their back-end. These jointly optimized hybrid (optical+digital) processors establish a new "diffractive language" between input electromagnetic waves that carry analog information and neural networks that process the digitized information at the back-end, providing the best of both worlds. Such hybrid designs can process spatially and temporally coherent, partially coherent, or incoherent input waves, providing universal coverage for any spatially varying set of point spread functions that can be optimized for a given task, executed in collaboration with digital neural networks. In this article, we highlight the utility of this exciting collaboration between engineered and programmed diffraction and digital neural networks for a diverse range of applications. We survey some of the major innovations enabled by the push-pull relationship between analog wave processing and digital neural networks, also covering the significant benefits that could be reaped through the synergy between these two complementary paradigms.