# Model-free Optical Processors using In Situ Reinforcement Learning with Proximal Policy Optimization

arXiv: https://arxiv.org/abs/2507.05583 · HTML: https://mlsanity.com/paper/2507.05583/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for energy, networks and operations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 31 of its 65 references that are in the map.
- First posted on arXiv 8 Jul 2025.

## Placed in

- Deep reinforcement learning methods / [RL for energy, networks and operations](https://mlsanity.com/deeprl/rl-for-energy-networks-and-operations/)

## Builds on (most cited first, 30 of 31)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Deep Image Prior](https://mlsanity.com/paper/1711.10925/) (2017)
- [Monte Carlo Gradient Estimation in Machine Learning](https://mlsanity.com/paper/1906.10652/) (2019)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [All-Optical Machine Learning Using Diffractive Deep Neural Networks](https://mlsanity.com/paper/1804.08711/) (2018)
- [Deep physical neural networks enabled by a backpropagation algorithm for arbitrary physical systems](https://mlsanity.com/paper/2104.13386/) (2021)
- [Training of Physical Neural Networks](https://mlsanity.com/paper/2406.03372/) (2024)
- [Training of photonic neural networks through in situ backpropagation](https://mlsanity.com/paper/1805.09943/) (2018)
- [Experimentally realized in situ backpropagation for deep learning in nanophotonic neural networks](https://mlsanity.com/paper/2205.08501/) (2022)
- [Wave Physics as an Analog Recurrent Neural Network](https://mlsanity.com/paper/1904.12831/) (2019)
- [Backpropagation-free Training of Deep Physical Neural Networks](https://mlsanity.com/paper/2304.11042/) (2023)
- [Misalignment Resilient Diffractive Optical Networks](https://mlsanity.com/paper/2005.11464/) (2020)
- [Reinforcement Learning in a large scale photonic Recurrent Neural Network](https://mlsanity.com/paper/1711.05133/) (2017)
- [Self-learning Machines based on Hamiltonian Echo Backpropagation](https://mlsanity.com/paper/2103.04992/) (2021)
- [L2ight: Enabling On-Chip Learning for Optical Neural Networks via Efficient in-situ Subspace Optimization](https://mlsanity.com/paper/2110.14807/) (2021)
- [Nonlinear Processing with Linear Optics](https://mlsanity.com/paper/2307.08533/) (2023)
- [Hybrid training of optical neural networks](https://mlsanity.com/paper/2203.11207/) (2022)
- [Hardware Beyond Backpropagation: a Photonic Co-Processor for Direct Feedback Alignment](https://mlsanity.com/paper/2012.06373/) (2020)
- [Trainable and Dynamic Computing: Error Backpropagation through Physical Media](https://mlsanity.com/paper/1407.6637/) (2014)
- [Multiplexed gradient descent: Fast online training of modern datasets on hardware neural networks without backpropagation](https://mlsanity.com/paper/2303.03986/) (2023)
- [Optical Generative Models](https://mlsanity.com/paper/2410.17970/) (2024)
- [Adaptive model selection in photonic reservoir computing by reinforcement learning](https://mlsanity.com/paper/2004.12575/) (2020)
- [Wave-based extreme deep learning based on non-linear time-Floquet entanglement](https://mlsanity.com/paper/2107.08564/) (2021)
- [Physical deep learning based on optimal control of dynamical systems](https://mlsanity.com/paper/2012.08761/) (2020)
- [Physics-inspired Neuroacoustic Computing Based on Tunable Nonlinear Multiple-scattering](https://mlsanity.com/paper/2304.08380/) (2023)
- [Model-free front-to-end training of a large high performance laser neural network](https://mlsanity.com/paper/2503.16943/) (2025)

## Built on by (most cited first, 2 of 2)

- [Snapshot 3D image projection using a diffractive decoder](https://mlsanity.com/paper/2512.20464/) (2025)
- [Wavelength-multiplexed massively parallel diffractive optical information storage and image projection](https://mlsanity.com/paper/2604.02624/) (2026)

## Abstract (arXiv)

Optical computing holds promise for high-speed, energy-efficient information processing, with diffractive optical networks emerging as a flexible platform for implementing task-specific transformations. A challenge, however, is the effective optimization and alignment of the diffractive layers, which is hindered by the difficulty of accurately modeling physical systems with their inherent hardware imperfections, noise, and misalignments. While existing in situ optimization methods offer the advantage of direct training on the physical system without explicit system modeling, they are often limited by slow convergence and unstable performance due to inefficient use of limited measurement data. Here, we introduce a model-free reinforcement learning approach utilizing Proximal Policy Optimization (PPO) for the in situ training of diffractive optical processors. PPO efficiently reuses in situ measurement data and constrains policy updates to ensure more stable and faster convergence. We experimentally validated our method across a range of in situ learning tasks, including targeted energy focusing through a random diffuser, holographic image generation, aberration correction, and optical image classification, demonstrating in each task better convergence and performance. Our strategy operates directly on the physical system and naturally accounts for unknown real-world imperfections, eliminating the need for prior system knowledge or modeling. By enabling faster and more accurate training under realistic experimental constraints, this in situ reinforcement learning approach could offer a scalable framework for various optical and physical systems governed by complex, feedback-driven dynamics.